Video segmentation method, server, storage medium and program product

By dividing the video data into multiple data segments and performing multimodal fusion feature extraction, the problem of inaccurate theme boundary recognition in the prior art is solved, and the accuracy and quality of video theme segmentation are improved.

CN119031206APending Publication Date: 2024-11-26ALIBABA (CHINA) CO LTD
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202410969075.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing video topic segmentation methods have inaccuracies when identifying theme boundaries, resulting in low accuracy and poor quality of video topic segmentation results, especially when the transition between topics is smooth or the correlation between them is high.

Method used

By dividing the video data to be divided into multiple data segments, extracting text features and visual features respectively, and performing multimodal fusion, predicting whether the data segment is the theme boundary, thereby accurately identifying the theme boundary of the video data and improving the theme boundary. The accuracy of recognition.

Benefits of technology

It improves the accuracy and quality of the video topic segmentation results, can better capture the relationships and interactions between different modes, and enhances the recognition ability of topic boundaries.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031206A_ABST
    Figure CN119031206A_ABST
Patent Text Reader

Abstract

The invention provides a video segmentation method, a server, a storage medium and a program product, and the method comprises the steps: segmenting to-be-segmented video data into a plurality of data segments, extracting the features of a single mode of each data segment through a video theme segmentation model, including the text features of a text mode and the visual features of a visual mode; the text features and the visual features of all the data fragments are fused, multi-modal information fusion can be carried out at the middle representation level, the relation and interaction between different modals can be better captured, and the multi-modal fusion features with higher quality of all the data fragments are obtained; furthermore, according to the multi-modal fusion feature of each data fragment, whether each data fragment is the theme boundary is predicted, the theme boundary of the video data can be accurately predicted, the accuracy of theme boundary identification is improved, and thus the accuracy and quality of a video theme segmentation result are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a video segmentation method, a server, a storage medium and a program product. Background Art

[0002] The surge in digital video content in recent years has highlighted the importance of efficient content navigation and understanding. Since the unstructured nature of videos poses a significant challenge to users seeking to quickly grasp or reference a specific topic, Video Topics Segmentation (VTS) has become an important tool to meet these needs. By dividing videos into coherent non-overlapping topical segments, VTS not only helps to intuitively understand the video content, but also enables rapid location and access to the video segments of interest. This is particularly important for further developing various video understanding tasks.

[0003] At present, the video topic segmentation method based on clustering algorithm obtains the category of each segment by inputting the learned visual features and text features of each segment into the clustering algorithm. However, the effect of the clustering process depends largely on the selected algorithm and related parameter configuration. In addition, clustering may be sensitive to input features and has difficulty in handling overlapping or blurred topic boundaries. In particular, when the transition between topics is smooth or the correlation is high, the clustering algorithm may have difficulty in accurately distinguishing different topics. Therefore, this video topic segmentation method has the problem of inaccurate topic boundary recognition, and the video topic segmentation results have low accuracy and poor quality. Summary of the invention

[0004] The present application provides a video segmentation method, server, storage medium and program product to solve the problem of inaccurate subject boundary recognition and the problem of low accuracy and poor quality of video subject segmentation results.

[0005] In a first aspect, the present application provides a video segmentation method, comprising: dividing video data to be segmented into multiple data segments, the data segments including text segments and video segments; inputting the multiple data segments into a video topic segmentation model, extracting text features and visual features of each of the data segments through the video topic segmentation model, fusing the text features and visual features of each of the data segments to obtain multimodal fusion features of each of the data segments, and predicting whether each of the data segments is a topic boundary based on the multimodal fusion features of each of the data segments to obtain the topic boundary of the video data; and segmenting the video data into multiple video segments based on the topic boundary of the video data, wherein adjacent video segments correspond to different topics.

[0006] In a second aspect, the present application provides a video segmentation method, comprising: obtaining a recorded conference video; dividing the conference video into multiple data segments, the data segments including text segments and video segments; inputting the multiple data segments into a video topic segmentation model, extracting text features and visual features of each of the data segments through the video topic segmentation model, and predicting whether each of the data segments is a topic boundary based on the text features and visual features of each of the data segments, thereby obtaining the topic boundary of the conference video; based on the topic boundary of the conference video, dividing the video data into multiple video segments, wherein adjacent video segments correspond to different topics; and outputting information of the multiple video segments of the conference video.

[0007] In a third aspect, the present application provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the server executes a method provided in any of the preceding aspects.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method provided in any of the aforementioned aspects is implemented.

[0009] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method provided in any of the aforementioned aspects.

[0010] The video segmentation method, server, storage medium and program product provided by the present application divide the video data to be segmented into multiple data segments, extract the single modal features of each data segment respectively through a video topic segmentation model, including text features of the text modality and visual features of the visual modality, and fuse the text features and visual features of each data segment. Multimodal information can be fused at the intermediate representation level, and the relationship and interaction between different modalities can be better captured to obtain higher quality multimodal fusion features of each data segment. Furthermore, based on the multimodal fusion features of each data segment, it is predicted whether each data segment is a topic boundary, so that the topic boundary of the video data can be accurately predicted, thereby improving the accuracy of topic boundary recognition, thereby improving the accuracy and quality of the video topic segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] Figure 1A schematic diagram of an exemplary system architecture applicable to the present application;

[0013] Figure 2 A flow chart of a video segmentation method provided for an exemplary embodiment of the present application;

[0014] Figure 3 An architecture diagram of a video topic segmentation model provided for an exemplary embodiment of the present application;

[0015] Figure 4 An exemplary architecture diagram of a multimodal fusion layer provided for an exemplary embodiment of the present application;

[0016] Figure 5 Another exemplary architecture diagram of a multimodal fusion layer provided for an exemplary embodiment of the present application;

[0017] Figure 6 An exemplary architecture diagram of a multimodal fusion layer including a MoE layer provided for an exemplary embodiment of the present application;

[0018] Figure 7 Another exemplary architecture diagram of a multimodal fusion layer including a MoE layer provided for an exemplary embodiment of the present application;

[0019] Figure 8 A flow chart of a method for training a video topic segmentation model provided for an exemplary embodiment of the present application;

[0020] Fig. 9 A training architecture diagram of a video topic segmentation model provided for an exemplary embodiment of the present application;

[0021] Fig.10 A flowchart of a video segmentation method provided for an exemplary embodiment of the present application;

[0022] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present application.

[0023] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0026] First, the terms involved in this application are explained:

[0027] Text Topic Segmentation (TTS): Segment a text sequence into a series of segments with non-overlapping topics and consistent topics within the segments.

[0028] Video Topic Segmentation (VTS): Segment a video into a series of segments with non-overlapping themes and consistent themes within the segments.

[0029] Multimodal Fusion (MF): Simultaneously modeling multimodal input data such as vision and text.

[0030] Coherence Modeling (CM): The process of ensuring logical and structural consistency between different data patterns or features in data analysis and processing.

[0031] Contrastive Learning (CL): Learn the feature representation of samples by comparing data with positive samples and negative samples in feature space.

[0032] Pretraining: Learning common language representations from large-scale corpora and providing a common model initialization method for various downstream tasks.

[0033] Continue Pretraining: Based on the pre-trained model, continue the pre-training process on unlabeled data related to downstream tasks to provide a better model initialization method for downstream tasks.

[0034] Finetuning: Fine-tuning the model based on the existing pre-trained model using data from the domain.

[0035] Visual question answering task: Given an input image and a question, determine the answer to the question from the visual information of the input image.

[0036] Image description task: Generate description text for an input image.

[0037] Visual entailment task: predict the semantic relevance of an input image and text, i.e., entailment, neutral, or contradiction.

[0038] Referential expression and comprehension task: locate the image area in the input image corresponding to the input text based on the input text.

[0039] Image generation task: Generate an image based on the input description text.

[0040] Text-based sentiment classification task: predict the sentiment classification information of the input text.

[0041] Text summarization task: Generate summary information of the input text.

[0042] Multimodal tasks: refers to downstream tasks whose input and output data involve multiple modal data such as images and text, such as visual question answering tasks, image description tasks, visual implication tasks, referential expression and understanding tasks, image generation tasks, etc.

[0043] Multimodal pre-trained model: refers to a pre-trained model whose input and output data involve multiple modal data such as images and text. After fine-tuning and training, it can be applied to multimodal task processing.

[0044] Pre-trained language model: A pre-trained model obtained by pre-training a large-scale language model (LLM).

[0045] A large model refers to a deep learning model with large-scale model parameters, usually containing hundreds of millions, tens of billions, or even hundreds of billions of model parameters. A large model can also be called a foundation model / foundation model (FM). It is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLMs) and multi-modal pre-training models.

[0046] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0047] In view of the problems of inaccurate theme boundary identification in existing video theme segmentation methods, and the low precision and poor quality of video theme segmentation results, the present application provides a video segmentation method, which divides video data to be segmented into multiple data segments, and the data segments include text segments and video segments; inputs the multiple data segments into a video theme segmentation model, extracts text features and visual features of each data segment through the video theme segmentation model, fuses the text features and visual features of each data segment to obtain multimodal fusion features of each data segment, and predicts whether each data segment is a theme boundary based on the multimodal fusion features of each data segment to obtain the theme boundary of the video data; and divides the video data into multiple video segments based on the theme boundary of the video data, wherein adjacent video segments correspond to different themes.

[0048] Among them, the video topic segmentation model is a pre-built, pre-trained and fine-tuned machine learning model. The video topic segmentation model includes a text encoding module, a visual encoding module, a multimodal fusion module and a classification prediction module. Among them, the text encoding module is used to extract the text features of the text segments in each data segment. The visual encoding module is used to extract the visual features of the video segments in each data segment. The multimodal fusion module is used to fuse the text features and visual features of multiple data segments to obtain the fused text features and fused visual features of each data segment. After the fused text features and fused visual features of each data segment are spliced, the multimodal fusion features of each data segment are obtained and input into the classification prediction module. The classification prediction module is used to classify and predict whether each data segment is a topic boundary based on the multimodal fusion features of each data segment to obtain the topic boundary of the video data.

[0049] The method of this embodiment extracts the features of a single modality of each data fragment respectively through a video topic segmentation model, including text features of the text modality and visual features of the visual modality, and then fuses the features of multiple modalities. Multimodal information can be fused at the intermediate representation level, and the relationship and interaction between different modalities can be better captured to obtain higher quality multimodal fusion features of each data fragment. Furthermore, based on the multimodal fusion features of each data fragment, it is possible to accurately predict whether each data fragment is a topic boundary, thereby improving the accuracy of topic boundary recognition, thereby improving the accuracy and quality of the video topic segmentation results.

[0050] In the embodiment of the present application, the video topic segmentation task is used as a segment-level sequence labeling task. The video data is divided into multiple data segments, and the multiple data segments are arranged in order to form a data segment sequence. The data segments that are the topic boundaries are predicted, that is, each data segment is labeled as a topic boundary or not a topic boundary. In this way, it can be determined which data segments are topic boundaries.

[0051] The topic boundary here is the last data segment of any topic. After knowing the last data segment of multiple topics, the video data is cut from the end of the data segment as the topic boundary to obtain multiple video segments. Each video segment includes one or more data segments, and the data segments in each video segment have the same topic. Adjacent video segments have different topics.

[0052] In addition, the topic boundary can also be defined as the first data segment of any topic. After the first data segments of multiple topics are known, the video data is cut from the beginning of the data segment as the topic boundary to obtain multiple video segments. Each video segment includes one or more data segments, and the data segments in each video segment have the same topic. Adjacent video segments have different topics.

[0053] Figure 1 FIG. 1 is a schematic diagram of an exemplary system architecture applicable to the present application. Figure 1 As shown, the system architecture includes a server and a terminal device. There is a communication link between the server and the terminal device, which can realize the communication connection between the server and the terminal device.

[0054] Among them, the server is a device with computing power deployed in the cloud or locally, such as a cloud cluster. The server stores a video topic segmentation model, which is used to classify and predict whether each data segment is a topic boundary according to the multiple data segments (including text segments and video segments) into which the video data is divided, and obtain a topic boundary prediction result. Based on the video data provided by the end-side device, the server divides the video data to be segmented into multiple data segments, predicts whether each data segment is a topic boundary through the video topic segmentation model, and obtains the topic boundary of the video data; according to the topic boundary of the video data, the video data is segmented into multiple video segments to achieve topic segmentation of the video data.

[0055] In addition, the server can also be responsible for pre-training and fine-tuning the video topic segmentation model to obtain a trained video topic segmentation model. Of course, the pre-training and fine-tuning of the video topic segmentation model can also be completed by other servers.

[0056] The end-side device can be an electronic device running a downstream application system, specifically a hardware device with network communication function, computing function and information display function, including but not limited to smart phones, tablet computers, desktop computers, local servers, cloud servers, etc. Video topic segmentation is required when the end-side device runs a downstream application system. The downstream application system run by the end-side device can be an online education system, a video conferencing assistant, etc. When implementing the corresponding system functions, it is necessary to perform video topic segmentation on the video data in the system.

[0057] For example, in an online education system, lecture videos involve multiple different topics, such as different knowledge points. By subject segmentation, the lecture videos can be segmented into multiple video segments according to different knowledge points. Adjacent video segments correspond to different knowledge points, which helps users quickly locate and access the video segments corresponding to the knowledge points of interest.

[0058] For example, in a video conference assistant, the generated conference videos involve multiple different topics, such as different conference topics. By subject segmentation of the conference video, the conference video can be segmented into multiple video segments according to different conference topics. Adjacent video segments correspond to different conference topics, which helps relevant personnel quickly locate and access the video segments corresponding to topics of interest.

[0059] based on Figure 1In the system architecture shown, when video topic segmentation is required, the terminal device sends the video data to be segmented to the server. The server divides the video data to be segmented into multiple data segments, which include text segments and video segments; the multiple data segments are input into the video topic segmentation model, and the text features and visual features of each data segment are extracted through the video topic segmentation model, and the text features and visual features of each data segment are fused to obtain the multimodal fusion features of each data segment, and the topic boundary of the video data; according to the topic boundary of the video data, the video data is segmented into multiple video segments, where adjacent video segments correspond to different topics.

[0060] In an example scenario, the server can return the subject segmentation result of the video data, that is, the segmentation information of multiple video segments of the video data, to the end-side device. The end-side device outputs the information of multiple video segments of the video data through the front-end interface, and the information of the video segment includes at least one of the following: timestamp, starting video frame, subject, and content summary. In response to the selection operation of any video segment, the end-side device starts playing the video data from the selected video segment.

[0061] In another example scenario, the server can output information of multiple video segments of video data through the front-end interface, and the information of the video segment includes at least one of the following: timestamp, starting video frame, theme, and content summary. In response to the selection operation of any video segment, the server starts playing the video data from the selected video segment.

[0062] The solution of this embodiment can be applied to various fields for video topic segmentation, including but not limited to application scenarios of online education and video conferencing assistants.

[0063] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0064] Figure 2 The flowchart of the video segmentation method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. Figure 2 As shown, the specific steps of this method are as follows:

[0065] Step S201: divide the video data to be segmented into multiple data segments, where the data segments include text segments and video segments.

[0066] Among them, the video data to be segmented can be video data generated by different application scenarios in various fields, such as lecture videos in online education systems, conference videos generated by video conferencing assistants, etc., and can specifically be video data in various formats. This embodiment does not specifically limit the source and format of the video data.

[0067] In this embodiment, the video topic segmentation is used as a sequence labeling task at the segment level, and the video data to be segmented is segmented into multiple segments (referred to as data segments). By predicting whether each data segment is a topic boundary (such as the last segment of a topic, or the first segment of a topic), the data segment that serves as the topic boundary in the segment sequence into which the video data is segmented is identified, that is, the topic boundary in the segment sequence into which the video data is segmented is determined, which is also referred to as the topic boundary of the video data. Furthermore, according to the video data topic boundary, the video data is segmented into multiple video segments, so that any video segment corresponds to a unique topic, and adjacent video segments correspond to different topics.

[0068] Usually, video data includes two parts, audio and video. In this step, when the video data is divided into multiple data segments, the audio of the video data is converted into text to obtain text information corresponding to the audio. The text information is divided into multiple text segments.

[0069] Specifically, automatic speech recognition (ASR) is performed on the audio of the video data. When the text information corresponding to the audio is obtained, the audio is input into the ASR system, which parses the audio and generates multiple possible candidate texts (usually containing multiple sentences and sentence timestamps). The candidate texts are given and sorted from high to low according to probability. The candidate text with the highest probability (i.e., "1-best") is the most accurate recognition result of the ASR system for the current input audio, and serves as the text information corresponding to the audio. In addition, speech-to-text methods other than ASR, such as speech-to-text methods based on machine learning, can also be used to convert the audio of the video data into corresponding text information, which is not specifically limited here.

[0070] When implementing the segmentation of text information into multiple text segments, the segmentation may be performed at the sentence granularity, where the text information is segmented into multiple sentences, each sentence being a text segment. Optionally, when segmenting the text information into multiple text segments, the segmentation may be performed at a coarser granularity (such as paragraph granularity or semantic block granularity) than the sentence granularity, where the text information is segmented into multiple coarser granularity text segments.

[0071] In this step, rule-based text segmentation, statistics-based text segmentation and machine learning-based text segmentation methods can be used to segment the text information into multiple text segments. This embodiment does not specifically limit the text segmentation method.

[0072] Further, according to the timestamps of the multiple text segments, the video data is segmented to obtain video segments corresponding to the text segments. Each text segment and the corresponding video segment constitute a data segment. Each of the segmented text segments contains one or more sentences. The timestamps of the text segments (including the start time and the end time) can be determined according to the timestamps of the sentences contained in the text segments.

[0073] In addition, when dividing the video data into multiple data segments, the existing video topic segmentation scheme that uses video topic segmentation as a segment-level sequence labeling task can also be used to segment the video data to be segmented, which is not specifically limited in this embodiment. For example, according to the voice activity detection method, the voice of the video data is segmented from the pause to obtain multiple voice segments, and the language segments are automatically converted into corresponding text segments by voice equipment, and the corresponding video segments are determined according to the timestamp of the voice segments. The text segment and video segment corresponding to each voice segment constitute a data segment.

[0074] For example, in the present application, n is used to represent the number of data segments into which the video data is segmented, and n data segments can be represented as in, Indicates the video segment contained in the i-th data segment, which includes k video frames It can be expressed as Represents the text segment contained in the i-th data segment. The text segment is a word sequence and can be expressed as Among them, ‖s i ‖ represents the number of words contained in the text segment contained in the i-th data segment. In addition, Add a feature token to the starting position (or end position) of A special marker indicating the start of insertion, such as "[BOS]", is used to mark the start of a text segment. Or, A special marker indicating the end of insertion, used to mark the end of a text segment.

[0075] Step S202: input multiple data fragments into the video topic segmentation model, extract the text features and visual features of each data fragment through the video topic segmentation model, fuse the text features and visual features of each data fragment to obtain multimodal fusion features of each data fragment, and predict whether each data fragment is a topic boundary based on the multimodal fusion features of each data fragment to obtain the topic boundary of the video data.

[0076] The video topic segmentation model is a pre-built, pre-trained and fine-tuned machine learning model. The video topic segmentation model includes a text encoding module, a visual encoding module, a multimodal fusion module and a classification prediction module.

[0077] In this embodiment, the video data is cut into multiple data segments and input into a video topic segmentation model. The text encoding module of the video topic segmentation model extracts the features of the text segments in each data segment to obtain the text features of each data segment, and the visual encoding module extracts the features of the video segments in each data segment to obtain the visual features of each data segment. The text features and visual features of each data segment are fused by the multimodal fusion module, and the multimodal information can be fused at the intermediate representation level, which can better capture the relationship and interaction between different modes and obtain the multimodal fusion features of each data segment. Through the classification prediction module, according to the multimodal fusion features of each data segment, classification prediction is performed on whether each data segment is a topic boundary. According to the classification prediction results, it can be determined which data segments are topic boundaries, thereby obtaining the topic boundaries of the video data.

[0078] Step S203: segment the video data into a plurality of video segments according to the subject boundaries of the video data, wherein adjacent video segments correspond to different subjects.

[0079] After obtaining the subject boundary of the video data, the video data is segmented from the subject boundary to obtain multiple video segments. Each video segment contains one or more data segments, and the data segments contained in the same video segment correspond to the same subject, that is, the same video segment corresponds to a unique subject. In addition, the multiple video segments obtained by segmentation are arranged in the order of timestamps, and adjacent video segments correspond to different subjects.

[0080] In an example scenario, after segmenting the video data into multiple video segments, the server may also generate information about the multiple video segments into which the video data is segmented, including but not limited to: generating a theme and / or content summary corresponding to each video segment, determining a timestamp of each video segment (including a start timestamp and / or an end timestamp), and determining a start frame of each video segment.

[0081] The server can output information of multiple video segments through the front-end interface, wherein the information of the video segment includes at least one of the following: timestamp, starting video frame, subject, and content summary.

[0082] Users can select the video segments they are interested in based on the subject, content summary and other information of each video segment, and quickly find the video segment they are interested in and play it by fast forwarding, rewinding, dragging the progress bar and other methods based on the timestamp and other information of the video segment they are interested in.

[0083] Optionally, the user can select the information of the video segment that he is interested in, such as clicking on the information of the video segment that he is interested in, to select the video segment that needs to be jumped to play. In response to the selection operation of any video segment, the server starts playing the video data from the selected video segment, so that the video segment that the user is interested in can be automatically located.

[0084] The method of this embodiment extracts the features of a single modality of each data fragment respectively through a video topic segmentation model, including text features of the text modality and visual features of the visual modality, and then fuses the features of multiple modalities. Multimodal information can be fused at the intermediate representation level, and the relationship and interaction between different modalities can be better captured to obtain higher quality multimodal fusion features of each data fragment. Furthermore, based on the multimodal fusion features of each data fragment, it is possible to accurately predict whether each data fragment is a topic boundary, thereby improving the accuracy of topic boundary recognition, thereby improving the accuracy and quality of the video topic segmentation results.

[0085] Figure 3 The following is an architecture diagram of a video topic segmentation model provided by an exemplary embodiment of the present application. Figure 3 As shown, the video topic segmentation model includes: a text encoding module, a visual encoding module, a multimodal fusion module and a classification prediction module.

[0086] The text encoding module is used to extract text features of the text segments in each data segment. The text encoding module can use a pre-trained text encoder, such as a BERT (Bidirectional Encoder Representations from Transformers) encoder, a Transformer encoder, a pre-trained language model, or other encoders that encode text information, which is not specifically limited in this embodiment.

[0087] The visual encoding module is used to extract the visual features of the video segments in each data segment. The visual encoding module can use a pre-trained visual encoder, such as a vision transformer (ViT), a convolutional neural network (CNN), a long short-term memory network (LSTM), a contrastive language-image pre-training (CLIP) model, etc., which is not specifically limited in this embodiment.

[0088] Optionally, in order to convert features of different modalities into the same dimension, a projection layer can be connected after the text encoder and the visual encoder respectively, for projecting the text features output by the text encoder and the visual features output by the visual encoder to the same dimension, so that the dimensions of the text features and visual features of each data segment obtained after the projection are consistent.

[0089] The multimodal fusion module is used to fuse the text features and visual features of multiple data segments to obtain fused text features and fused visual features of each data segment.

[0090] The multimodal fusion module may include a plurality of multimodal fusion layers stacked in sequence (e.g. Figure 3 The "M×" shown in the figure indicates that M multimodal fusion layers are stacked in sequence. Each multimodal fusion layer is used to fuse the text features and visual features of multiple data segments input to the layer, and output the fused text features and fused visual features of each data segment. The fused text features and fused visual features of each data segment output by the last multimodal fusion layer are used as the fused text features and fused visual features of each data segment.

[0091] Furthermore, after the fused text features and fused visual features of each data segment are spliced, the multimodal fusion features of each data segment are obtained and input into the classification prediction module.

[0092] The classification prediction module is used to classify and predict whether each data segment is a topic boundary according to the multimodal fusion features of each data segment, and obtain the topic boundary of the video data. Among them, the classification prediction module can use a binary classification predictor to predict the probability that each data segment is a topic boundary. The prediction result output by the classification prediction module includes the probability that each data segment is a topic boundary. Furthermore, a data segment whose probability of a topic boundary is greater than or equal to a probability threshold is regarded as a topic boundary, while a data segment whose probability of a topic boundary is less than a probability threshold is not a topic boundary.

[0093] The probability threshold may be configured and adjusted according to the requirements of the actual application scenario. For example, the probability threshold may be configured to be 0.5 or other values, which are not specifically limited here.

[0094] based on Figure 3 The video topic segmentation model shown in the above step S202 extracts the text features and visual features of each data segment through the video topic segmentation model, fuses the text features and visual features of each data segment, obtains the multimodal fusion features of each data segment, and predicts whether each data segment is a topic boundary according to the multimodal fusion features of each data segment to obtain the topic boundary prediction result, which is specifically achieved by the following method:

[0095] The text features of the text segments in each data segment are extracted through the text encoding module; the visual features of the video segments in each data segment are extracted through the visual encoding module. Furthermore, the text features and visual features of multiple data segments are fused through the multimodal fusion module to obtain the fused text features and fused visual features of each data segment. The fused text features and fused visual features of each data segment are spliced ​​to obtain the multimodal fusion features of each data segment. Furthermore, through the classification prediction module, based on the multimodal fusion features of each data segment, classification prediction is performed on whether each data segment is a topic boundary to obtain the topic boundary of the video data.

[0096] For example, Figure 3 As shown, the text segment in the n data segments into which the video data is segmented The input text encoding module encodes the n text fragments through the text encoding module, extracts and outputs the text features of each text fragment. The output features of the text encoding module are projected to a unified dimension through the projection layer to obtain the text features of each data fragment. The text features of the n data fragments can be expressed as: {t1,…,t n}.

[0097] The video data is divided into n data segments of the video segment The visual encoding module is used to encode the n video clips and extract the visual features of each video clip. The output features of the visual encoding module are projected to a unified dimension through the projection layer to obtain the visual features of each data clip. The visual features of the n data clips can be expressed as: {v1,…,v n}.

[0098] Optionally, due to the high computational complexity of the visual coding module, the video segments in each data segment can be separately input into the visual coding module for encoding; or, the n video segments can be divided into multiple groups, each group containing a smaller number (compared to the total number n) of video segments, and a smaller number of video segments in a group are input into the visual coding module for encoding each time.

[0099] Furthermore, the text features of n data segments {t1,…,t n} and visual features {v1,…,v n} Input the multimodal fusion module (including M multimodal fusion layers stacked in sequence). The multimodal fusion module is used to fusion the text features {t1,…,t n} and visual features {v1,…,v n} to obtain the fused text features of n data fragments and fusion of visual features

[0100] The fused text features and fused visual features of each data segment are spliced ​​together to obtain the multimodal fusion features of each data segment. The multimodal fusion features of n data segments are represented as {m1,…,m n}.

[0101] Furthermore, the multimodal fusion features of n data segments {m1,…,m n} Input the classification prediction module, and the classification prediction module predicts the probability that each data segment is a topic boundary, which is expressed as {p1,…,p n}.

[0102] Furthermore, according to the probability {p1,…,p n}, for any data segment i, if the probability p of the data segment being a topic boundary i If the probability p of the data segment is greater than or equal to the probability threshold, the data segment is determined to be a topic boundary. i If the probability is less than the threshold, it is determined that the data segment is not a topic boundary.

[0103] The video topic segmentation model provided in this embodiment extracts the text features and visual features of each data segment respectively through the text encoding module and the visual encoding module to obtain multiple single modal features. Then, the features of multiple modalities are fused through the multimodal fusion module, and the multimodal information can be fused at the intermediate representation level, which can better capture the relationship and interaction between different modalities and obtain higher quality multimodal fusion features of each data segment. Furthermore, through the classification prediction module, according to the multimodal fusion features of each data segment, it is possible to accurately predict whether each data segment is a topic boundary, thereby improving the accuracy of topic boundary recognition, thereby improving the accuracy and quality of the video topic segmentation results.

[0104] In an optional embodiment, the multimodal fusion module may include an attention layer and a mapping layer, and the attention layer performs attention calculation on the text features and visual features of multiple data fragments to obtain a multimodal fusion representation of each data fragment. Through the mapping layer, the multimodal fusion representation of each data fragment is mapped into a fused text feature and a fused visual feature.

[0105] The attention layer can be implemented by a self-attention module or a neural network combining self-attention and cross-attention. The mapping layer can be implemented by a feedforward neural network (FNN) or a multilayer perceptron (MLP).

[0106] For example, the multimodal fusion module can be implemented based on the Merge-Attention mechanism. Figure 4 As shown, the multimodal fusion layer in the multimodal fusion module can be implemented by a merged attention layer. The merged attention layer includes a self-attention layer and a feedforward neural network layer. The merged results of the text features and visual features of multiple data fragments are respectively used as query features, key features and value features, and input into the self-attention layer of the first merged attention layer in the multimodal fusion module for self-attention calculation to obtain a multimodal fusion representation of multiple data fragments; the multimodal fusion representation of multiple data fragments is input into the feedforward neural network layer, and the multimodal fusion representation of multiple data fragments is mapped into text features and visual features through the feedforward neural network layer to obtain the fused text features and fused visual features of multiple data fragments. The fused text features and fused visual features output by the last multimodal fusion layer of the multimodal fusion module are used as the fused text features and fused visual features finally output by the multimodal fusion module.

[0107] Taking a multimodal fusion layer in the multimodal fusion module as an example, its input is the fused text features output by the previous multimodal fusion layer (represented by l-1) And the fused visual features The combined result of the input fused text features and the fused visual features is The merged results are used as query features (Q v;t Indicated by), key features (K in the figure v;t denoted) and value features (denoted by V in the figure) v;t denoted by) inputs the self-attention layer of the current multimodal fusion layer (denoted by l) for self-attention calculation to obtain the multimodal fusion representation of n data fragments, which can be expressed as Furthermore, through the feedforward neural network layer, the multimodal fusion representation of n data segments is Mapping to text features and visual features The fused text features and fused visual features output by the current multimodal fusion layer (represented by l). The fused text features and fused visual features output by the last multimodal fusion layer are used as the final fused text features. and fusion of visual features

[0108] For example, the multimodal fusion module can be implemented based on the Co-Attention mechanism. Figure 5 As shown, the multimodal fusion layer in the multimodal fusion module can be implemented by using a collaborative attention layer. Each collaborative attention layer includes two branches, each of which includes a self-attention layer, a cross-attention layer, and a feedforward neural network layer.

[0109] The text features of multiple data fragments are respectively used as the input query features, input key features and input value features of the self-attention layer in one of the branches (denoted as the first branch), and are input into the self-attention layer of the first branch for self-attention calculation; the visual features of multiple data fragments are respectively used as the input query features, input key features and input value features of the self-attention layer in another branch (denoted as the second branch), and are input into the self-attention layer of the second branch for self-attention calculation. Figure 5 Q t , K t and V t Q represents the query feature, key feature, and value feature in the first branch (the branch corresponding to the text feature). v , K v and V v Represents the query features, key features, and value features in the second branch (the branch corresponding to the visual features).

[0110] Furthermore, the output query features of the self-attention layer of the first branch are used as the input query features of the cross-attention layer in the first branch, and the output key features and output value features of the self-attention layer of the second branch are used as the input key features and input value features of the cross-attention layer in the first branch, and are input into the cross-attention layer of the first branch for cross-attention calculation to obtain the multimodal fusion representation of the first branch.

[0111] The output query features of the self-attention layer of the second branch are used as the input query features of the cross-attention layer in the second branch. The output key features and output value features of the self-attention layer of the first branch are used as the input key features and input value features of the cross-attention layer in the second branch, respectively. They are input into the cross-attention layer of the second branch for cross-attention calculation to obtain the multimodal fusion representation of the second branch.

[0112] Furthermore, the multimodal fusion representation of the first branch is input into the feedforward neural network layer of the first branch, and the multimodal fusion representation of the first branch is mapped to text features through the feedforward neural network layer of the first branch to obtain fused text features of multiple data segments.

[0113] The multimodal fusion representation of the second branch is input into the feedforward neural network layer of the second branch, and the multimodal fusion representation of the second branch is mapped into visual features through the feedforward neural network layer of the second branch to obtain fused visual features of multiple data segments.

[0114] The fused text features and fused visual features output by the last multimodal fusion layer of the multimodal fusion module serve as the fused text features and fused visual features finally output by the multimodal fusion module.

[0115] In an optional embodiment, the multimodal fusion module may include an attention layer and a Mixture of Experts (MoE) layer. Through the attention layer, the text features and visual features of multiple data segments are subjected to attention calculation to obtain a multimodal fusion representation of each data segment. Through the mixed expert layer, based on the mixed expert mechanism, at least one expert module is selected to map the multimodal fusion representation of each data segment into fused text features and fused visual features.

[0116] Each multimodal fusion layer in the multimodal fusion module includes an attention layer and a hybrid expert. The attention layer can be implemented by a self-attention module or a neural network combining self-attention and cross-attention. Figure 4 or Figure 5 The attention layer shown in . The hybrid expert layer includes: a gating module, a merging module, and multiple expert modules.

[0117] Through the gating module, at least one expert module is selected according to the multimodal fusion representation of each input data segment. Through at least one expert module, the multimodal fusion representation of each data segment is mapped into fused text features and fused visual features. Through the merging module, according to the weight of at least one expert module, the output of at least one expert module is merged (weighted sum or weighted average) to obtain the fused visual features of each data segment.

[0118] Among them, each expert module is a mapping module, which can be implemented using a feedforward neural network, a multilayer perceptron, etc.

[0119] The gating module can be a gating network based on a neural network (such as a convolutional layer, a fully connected layer), which is responsible for receiving input data and calculating the weight of each expert module based on the input data. These weights are usually normalized by a softmax function to ensure that the sum of the weights of all expert modules is 1, thereby forming a probability distribution. Optionally, the gating module can also be implemented using other methods such as a decision tree or random forest method, which is not specifically limited in this embodiment.

[0120] Optionally, in order to further improve computational efficiency, a sparse gating mechanism can be introduced into the MoE layer to ensure that only a small number of expert modules are activated to process input data. For example, a Top-k gating mechanism can be used to select k expert modules with higher weights for activation based on the weights of each expert module output by the gating module. This approach can significantly reduce the amount of computation while maintaining the expressiveness of the MoE module. For example, a threshold selection mechanism can be used to activate only expert modules with weights greater than a weight threshold based on the weights of each expert module determined by the gating module.

[0121] The merging module is responsible for fusing the outputs of each expert module to obtain the final output of the hybrid expert layer. The merging module ensures that the MoE module can integrate the capabilities of each expert module while maintaining the consistency and coherence of the overall output.

[0122] For example, the multimodal fusion module is implemented by combining the hybrid expert mechanism with the merged attention mechanism. Figure 6As shown, the multimodal fusion layer in the multimodal fusion module includes: a self-attention layer and a mixed expert layer (i.e., MoE layer). The merged results of the text features and visual features of multiple data fragments are respectively used as query features, key features, and value features, and input into the self-attention layer of the first layer of the merged attention layer in the multimodal fusion module for self-attention calculation to obtain the multimodal fusion representation of the multiple data fragments; the multimodal fusion representation of the multiple data fragments is input into the mixed expert layer, and the mixed expert layer selects to use at least one expert module based on the mixed expert mechanism to map the multimodal fusion representation of each data fragment into fused text features and fused visual features. The fused text features and fused visual features output by the last layer of the multimodal fusion layer of the multimodal fusion module are used as the fused text features and fused visual features finally output by the multimodal fusion module.

[0123] For example, Figure 6 A schematic diagram of the structure of the multimodal fusion layer provided in this embodiment; the multimodal fusion layer used in this embodiment can be obtained by Figure 4 The feed-forward neural network layer in the multimodal fusion layer shown is replaced by a mixture of experts (i.e., MoE) layer implementation. Figure 6 The implementation principle of the self-attention layer shown in Figure 4 The implementation principle of the self-attention layer is the same as that of the above embodiment, and the steps are not repeated here. Figure 6 As shown, the multimodal fusion representation of the n data fragments output by the self-attention layer is input into the gating module of the MoE layer. The gating module selects and activates at least one expert module from all s expert modules according to the multimodal fusion representation of each input data fragment, and provides the weights of each activated expert module to the merging module. Each activated expert module participates in this calculation, and the multimodal fusion representation of each data fragment is mapped to fused text features and fused visual features. The merging module performs weighted summation of the output results of each expert module according to the weight of at least one expert module to obtain the fused text features and fused visual features of each data fragment.

[0124] For example, Figure 7 FIG. 1 is a schematic diagram of another multimodal fusion layer provided in this embodiment. The multimodal fusion layer used in this embodiment can be Figure 5 The two feed-forward neural network layers in the multimodal fusion layer shown are replaced by a mixture of experts (MoE) layer implementation. Figure 7 The implementation principle of the collaborative attention layer shown in Figure 5 The collaborative attention layer in the embodiment is implemented in the same principle, please refer to the relevant content of the above embodiment for details, and the steps are repeated here. Figure 7As shown, the multimodal fusion representation of the first branch and the multimodal fusion representation of the second branch in the collaborative attention layer are input to the gating module of the MoE layer. The gating module selects and activates at least one expert module based on the multimodal fusion representation of each input data segment, and provides the weights of each activated expert module to the merging module. Each activated expert module participates in this calculation, and maps the multimodal fusion representation of each data segment to fused text features and fused visual features. The merging module performs weighted summation of the output results of each expert module based on the weight of at least one expert module to obtain the fused text features and fused visual features of each data segment.

[0125] In this embodiment, by introducing the MoE mechanism in the multimodal fusion layer, the multimodal fusion representation is input to the MoE layer for processing. The MoE layer can promote the fusion of deeper cross-modal information, improve the model capacity without increasing the computational complexity in proportion to the increased capacity, and reduce the computational complexity to a certain extent, thereby optimizing the multimodal fusion process.

[0126] The video topic segmentation model in the aforementioned embodiment can be obtained by constructing an initial video topic segmentation model and performing supervised training on the video topic segmentation model using a training set with annotated topic boundaries of a downstream task.

[0127] When constructing the initial video topic segmentation model, a pre-trained neural network can be used. Specifically, the text encoding module and the visual encoding module can be implemented using a pre-trained text encoder and a pre-trained visual encoder, and the classification predictor can be implemented using a pre-trained classifier. The parameters of the multimodal fusion module can be determined by random initialization.

[0128] In order to better initialize the parameters of the multimodal fusion module in the video topic segmentation model, unlabeled video data can be used for pre-training (continuous pre-training here) before using the training set with annotated topic boundaries for supervised training, and then the training set with annotated topic boundaries is used for fine-tuning training.

[0129] Figure 8 A flow chart of a training method for a video topic segmentation model provided by an exemplary embodiment of the present application. Figure 8 As shown, the specific process of this method is as follows:

[0130] Step S800: Obtain an initial video topic segmentation model.

[0131] In this embodiment, the architecture of the initial video topic segmentation model constructed is as follows: Figure 3As shown, it includes a text encoding module, a visual encoding module, a multimodal fusion module and a classification prediction module. Among them, the parameters of the text encoding module can be initialized using the parameters of the pre-trained text encoder. The parameters of the visual encoding module can be initialized using the parameters of the pre-trained visual encoder. The parameters of the multimodal fusion module are obtained by random initialization. The parameters of the classification predictor can be initialized using the parameters of the pre-trained classifier, or can be randomly initialized.

[0132] In the pre-training stage and the fine-tuning stage, the parameters of the entire video topic segmentation model can be adjusted. In addition, in order to improve the training speed, in the pre-training stage and the fine-tuning stage, the parameters of the visual encoding module and / or the text encoding module are frozen (fixed), and the parameters of the remaining modules are adjusted. In a preferred embodiment, the parameters of the visual encoding module can be frozen, and the parameters of the text encoding module, the multimodal fusion module, and the classification prediction module can be adjusted.

[0133] Step S801: In the pre-training stage, video samples with unknown subject boundaries are used to construct video samples with pseudo-annotated subject boundaries, and the video samples with pseudo-annotated subject boundaries are used to train the initial video subject segmentation model to obtain a pre-trained video subject segmentation model.

[0134] In this embodiment, in the pre-training stage, an initial video topic segmentation model is trained using unlabeled video samples to obtain a pre-trained video topic segmentation model.

[0135] Specifically, firstly, video samples with unknown subject boundaries are used to construct video samples with pseudo-annotated (automatically generated) subject boundaries. Then, the video samples with pseudo-annotated subject boundaries are used to supervise the initial video subject segmentation model to obtain a pre-trained video subject segmentation model.

[0136] When constructing a video sample with pseudo-annotated subject boundaries, the video sample with unknown subject boundaries is divided into multiple segments, each of which is called a segmented segment; for any video sample with unknown subject boundaries, at least one segmented segment that does not belong to the video sample is inserted into the video sample, and / or at least one segmented segment of the video sample is replaced with a segmented segment that does not belong to the video sample, a new video sample is constructed, and the pseudo-annotated subject boundaries of the new video sample are determined.

[0137] In a new video sample, the newly inserted segment has a different theme from the adjacent segment, and the replaced segment has a different theme from the adjacent segment. By discontinuously inserting (or replacing) segments of other video samples into the video sample, the inserted (or replaced) segment has a different theme from the adjacent segment, thereby constructing a definite theme boundary.

[0138] For the constructed new video sample, the video sample is divided into multiple sample segments. For any two adjacent segments with different themes in the new video sample, the last sample segment in the previous segment (or the first sample segment in the next segment) is used as the subject boundary. In this way, the pseudo-annotated subject boundary of the new video sample can be determined. Since the subject boundary is automatically generated instead of being obtained by marking the video sample, the constructed subject boundary is called the pseudo-annotated subject boundary here.

[0139] Optionally, when implementing the process of dividing a video sample with unknown subject boundaries into multiple segments, the duration distribution of the subject is obtained; for any video sample with unknown subject boundaries, a duration is sampled according to the duration distribution, and the video sample is divided into multiple segments according to the sampled duration.

[0140] Among them, the duration distribution of the theme can be obtained by counting and analyzing the duration of the video segments of different themes in the video samples with marked theme boundaries. Exemplarily, the kernel density estimation (KDE) model or other statistical methods can be used to approximate the duration distribution of each theme in the video samples with marked theme boundaries. For example, the duration range of the theme can be determined according to the minimum duration and maximum duration of each theme in the video sample, and the duration range of the theme is divided into multiple intervals as the duration interval of the theme. According to the duration of the video segments of each theme in the video sample, the proportion of the number of video segments falling into each duration interval is counted (as the sampling probability of the duration interval of the corresponding theme), and the duration distribution of the theme is obtained.

[0141] Specifically, when any video sample is divided into multiple segments, the proportion of the number of video segments falling into the duration interval of each theme can be used as the sampling probability of each duration interval according to the duration distribution of the theme, and a duration interval is sampled, and a value is randomly sampled from the duration interval as the duration, and the video sample is divided into multiple segments according to the sampled duration.

[0142] Optionally, when implementing the segmentation of a video sample with unknown subject boundaries into multiple segments, a duration may be randomly sampled from a pre-configured subject duration range, and the video sample may be segmented into multiple segments according to the sampled duration.

[0143] Furthermore, when constructing a new video sample, for any video sample with unknown subject boundaries, for each segment in the video sample, one of the following three operations is performed with equal probability: inserting a segment randomly selected from other video samples before or after the current segment, replacing it with a segment randomly selected from other video samples, or not making any changes. In this way, at least one segment that does not belong to the video sample can be inserted into the video sample, and / or at least one segment of the video sample can be replaced with a segment that does not belong to the video sample to construct a new video sample.

[0144] After obtaining the video samples with pseudo-annotated subject boundaries, in the pre-training stage, the process of training the initial video subject segmentation model using the video samples with pseudo-annotated subject boundaries is as follows:

[0145] A video sample with pseudo-annotated subject boundaries is divided into multiple sample segments, and the sample segments include text segments and video segments; the multiple sample segments of the video sample are input into a video subject segmentation model to be trained, and the text features and visual features of each sample segment are extracted through the video subject segmentation model to be trained, and the text features and visual features of each sample segment are fused to obtain multimodal fusion features of each sample segment, and based on the multimodal fusion features of each sample segment, it is predicted whether each sample segment is a subject boundary, so as to obtain a subject boundary prediction result of the video sample.

[0146] The cross entropy loss is calculated based on the topic boundary prediction results of the video samples and the topic boundaries of the video samples.

[0147] For example, when the topic boundary is defined as the last data segment in the topic, the last data segment of the video sample is not calculated for loss. The cross entropy loss is calculated by the following formula (1):

[0148]

[0149] Among them, l vts represents the cross entropy loss, p i represents the probability that the i-th data segment is a boundary topic. i The label indicating whether the i-th data segment is a topic boundary, y i =1 means that the i-th data segment is the topic boundary, y i = 0 indicates that the i-th data segment is not a topic boundary. ln indicates the natural logarithm. n indicates the number of data segments.

[0150] In addition, the calculation method of the cross entropy loss function can be modified based on formula (1), for example, ln is replaced by logarithm log with base 10, or other cross entropy loss calculation methods can be used, which are not specifically limited here.

[0151] Furthermore, according to the cross entropy loss, the parameters of the video topic segmentation model to be trained are adjusted through back propagation.

[0152] In an optional embodiment, in the pre-training stage, when the text features and visual features of each sample fragment are fused through a multimodal fusion module to obtain the multimodal fusion features of each sample fragment, the text features and visual features of multiple data fragments are first fused through the multimodal fusion module to obtain the fused text features and fused visual features of each data fragment; then the fused text features and fused visual features of each data fragment are spliced ​​to obtain the multimodal fusion features of each data fragment.

[0153] In the pre-training stage, a cross-modal alignment task is introduced, and contrastive learning loss is used to adjust the features learned by the multimodal fusion module in the model to maximize the similarity between the fused text features and the fused visual features of the same data segment, while reducing the similarity of the modal features between different data segments.

[0154] The pre-training stage also includes: calculating the modal contrast learning loss based on the fused text features and fused visual features of each sample segment of the video sample, and the modal contrast learning loss is used to increase the similarity between the fused text features and the fused visual features of the same sample segment, and reduce the similarity between the fused text features and the fused visual features of different sample segments.

[0155] For example, in the pre-training stage, the following formula (2) can be used to calculate the modality contrast learning loss based on the fused text features and fused visual features of each sample segment of the video sample:

[0156]

[0157] Among them, l cma represents the modality contrast learning loss, n represents the number of data segments into which the video sample is cut, They represent the fused visual features and fused text features of the i-th data segment respectively, Represents the fused visual features of the i-th data segment and fusion text features The similarity. represents the fused text features of the jth data segment, Represents the fused visual features of the i-th data segment Fusion text features with the jth data segment ∈ is a pre-configured hyperparameter. In order to prevent the denominator from being 0, the specific value can be configured according to the needs of the actual application scenario. For example, ∈ can be set to 0.1.

[0158] In addition, the calculation method of modality contrast learning loss can be modified based on formula (2), for example, Replace with Will Replace with Or it can be implemented by other cross entropy loss calculation methods, which are not specifically limited here.

[0159] Among them, the similarity between the fused visual features and the fused text features of the data segment can be achieved by calculating the Euclidean distance, cosine similarity, or correlation coefficient between the fused visual features and the fused text features, which is not specifically limited in this embodiment.

[0160] Furthermore, according to the modality contrast learning loss, the parameters of the video topic segmentation model to be trained are adjusted through back propagation, mainly the parameters of the multimodal fusion module.

[0161] For example, in the pre-training stage, the cross entropy loss l vts and the modality contrastive learning loss l cma The weighted sum is used to obtain the comprehensive loss. Based on the comprehensive loss, the parameters of the video topic segmentation model to be trained are adjusted through back propagation. Among them, the cross entropy loss l vts and the modality contrastive learning loss l cma The weight coefficient can be configured and adjusted according to the needs of the actual application scenario and is not specifically limited here.

[0162] In this embodiment, by adding modality contrast learning loss in the pre-training stage, a cross-modal alignment task customized for the downstream video topic segmentation task is introduced, unlabeled data is used to learn pseudo-labeled topic boundaries, and the alignment of different modality features is enhanced.

[0163] In an optional embodiment, the multimodal fusion module of the video topic segmentation model includes an attention layer and a mixed expert layer. The pre-training stage also includes: calculating the balance loss of the mixed expert layer (i.e., l balance ); According to the balance loss, adjust the parameters of the mixed expert layer. By adding the balance loss function, the load imbalance problem between experts in the model training process can be solved, which helps each expert module to be fully trained, avoids some expert modules being overused while other expert modules are relatively idle, and avoids this imbalance affecting the training efficiency and having a negative impact on the final performance of the model.

[0164] The balanced loss function of the hybrid expert layer consists of two parts: the importance loss function (i.e., importance ) and the load loss function (i.e., l load). Among them, the importance loss function is used to define the importance of each expert module, which is usually reflected by measuring the number of samples processed by each expert module. The purpose of the importance loss function is to encourage all expert modules to participate in training and avoid some expert modules being ignored because they process fewer samples. By adding the importance loss function, the mixed expert layer will tend to distribute samples more evenly to each expert module, thereby improving the overall training effect. The common way to calculate the load loss function is to calculate the difference between the number of samples processed by each expert and the average number of samples, and penalize it.

[0165] The load loss function is used to solve the problem of unbalanced sample weights between expert modules. The purpose of the load loss function is to ensure that each expert module receives relatively balanced sample weights, to avoid some expert modules being underestimated because they receive a large number of samples with small weights, while other expert modules are overused because they receive a small number of samples with large weights. The common way to calculate the load loss function is to calculate the variance of the sample weights received by the expert module.

[0166] The balanced loss function of the mixed expert layer is the importance loss function (i.e., l importance ) and the load loss function (i.e., l load ), that is, l balance = l importance +l load In this embodiment, the balance loss function of the hybrid expert layer is importance The balanced loss function of any existing hybrid expert model can be used for implementation, and is not specifically limited here.

[0167] For example, the cross entropy loss l can be calculated during the pre-training phase. vts And the balance loss function l balance , for the cross entropy loss l vts And the balance loss function l balance The weighted sum is used to obtain the comprehensive loss. Based on the comprehensive loss, the parameters of the video topic segmentation model to be trained are adjusted through back propagation. Among them, the cross entropy loss l vts And the balance loss function l balance The weight coefficient can be configured and adjusted according to the needs of the actual application scenario and is not specifically limited here.

[0168] For example, the cross entropy loss l can be calculated during the pre-training phase. vts , modality contrastive learning loss l cma And the balance loss function l balance , for the cross entropy loss l vts , modality contrastive learning loss l cma And the balance loss function l balanceThe weighted summation gives the comprehensive loss in the pre-training phase, i.e., l pretrain = l vts +αl cma +βl balance Among them, α and β are the modality contrast learning loss l cma And the balance loss function l balance The weight coefficients α and β are both greater than 0 and less than 1, which can be configured and adjusted according to the needs of the actual application scenario and are not specifically limited here. Further, according to the comprehensive loss l in the pre-training stage pretrain , adjust the parameters of the video topic segmentation model to be trained through back propagation.

[0169] After the pre-training stage is completed, a pre-trained video topic segmentation model is obtained. Through the pre-training stage, the parameters of the video topic segmentation model can be better initialized, especially better initialization parameters of the multimodal fusion module can be obtained.

[0170] Step S802: In the fine-tuning stage, the pre-trained video topic segmentation model is trained using video samples with marked topic boundaries to obtain a trained video topic segmentation model.

[0171] In the fine-tuning stage, the process of training the pre-trained video topic segmentation model using video samples with annotated topic boundaries is as follows:

[0172] A video sample with marked topic boundaries is divided into multiple sample segments, and the sample segments include text segments and video segments; the multiple sample segments of the video sample are input into a pre-trained video topic segmentation model, and the text features and visual features of each sample segment are extracted through the pre-trained video topic segmentation model, and the text features and visual features of each sample segment are fused to obtain multimodal fusion features of each sample segment, and based on the multimodal fusion features of each sample segment, it is predicted whether each sample segment is a topic boundary, so as to obtain a topic boundary prediction result of the video sample.

[0173] The cross entropy loss is calculated based on the prediction results of the topic boundaries of the video samples and the annotated topic boundaries of the video samples. According to the cross entropy loss, the parameters of the video topic segmentation model to be trained are adjusted through back propagation. The calculation method of the cross entropy loss in the fine-tuning stage is similar to that in the pre-training stage, and will not be repeated here.

[0174] In an optional embodiment, in the fine-tuning stage, when the text features and visual features of each sample fragment are fused through a multimodal fusion module to obtain the multimodal fusion features of each sample fragment, the text features and visual features of multiple data fragments are first fused through the multimodal fusion module to obtain the fused text features and fused visual features of each data fragment; then the fused text features and fused visual features of each data fragment are spliced ​​to obtain the multimodal fusion features of each data fragment.

[0175] In the fine-tuning stage, a cross-modal alignment task is introduced that is the same as that in the pre-training stage. Contrastive learning loss is used to adjust the features learned by the multimodal fusion module in the model to maximize the similarity between the fused text features and the fused visual features of the same data segment, while reducing the similarity between the modal features of different data segments. The cross-modal alignment task in the fine-tuning stage is the same as that in pre-training. This continuity ensures that the modalities remain consistent during pre-training and fine-tuning, thereby promoting consistent interactions between different modalities.

[0176] In the fine-tuning stage, it also includes: calculating the modal contrast learning loss according to the fused text features and fused visual features of each sample segment of the video sample, and the modal contrast learning loss is used to increase the similarity between the fused text features and fused visual features of the same sample segment, and reduce the similarity between the fused text features and fused visual features of different sample segments. The calculation method of the modal contrast loss in the fine-tuning stage is similar to that in the pre-training stage, and will not be repeated here.

[0177] Furthermore, according to the modality contrast learning loss, the parameters of the video topic segmentation model to be trained are adjusted through back propagation, mainly the parameters of the multimodal fusion module.

[0178] For example, in the fine-tuning stage, the cross entropy loss and the modality contrast learning loss are weighted and summed to obtain a comprehensive loss, and the parameters of the video topic segmentation model to be trained are adjusted by back propagation according to the comprehensive loss. Among them, the weight coefficients of the cross entropy loss and the modality contrast learning loss can be configured and adjusted according to the needs of the actual application scenario, and are not specifically limited here.

[0179] In this embodiment, by adding a modality contrast learning loss similar to that in the pre-training stage in the fine-tuning stage, a cross-modal alignment task customized for the downstream video topic segmentation task is introduced, so that the model learns to mark topic boundaries and enhances the alignment of different modal features.

[0180] In an optional embodiment, the multimodal fusion module of the video topic segmentation model includes an attention layer and a mixed expert layer. The fine-tuning stage also includes: calculating the balance loss of the mixed expert layer (i.e., l balance); According to the balance loss, adjust the parameters of the mixed expert layer. By adding the balance loss function, the problem of load imbalance between experts during model training can be solved, which helps each expert module to be fully trained, avoids some expert modules being overused while other expert modules are relatively idle, and thus avoids such imbalance affecting the training efficiency and having a negative impact on the final performance of the model. The calculation method of the balance loss of the mixed expert layer in the fine-tuning stage is similar to that in the pre-training stage, and will not be repeated here.

[0181] Exemplarily, in the fine-tuning stage, the cross entropy loss, modality contrast learning loss and balance loss function can be calculated, and the cross entropy loss, modality contrast learning loss and balance loss function are weighted and summed to obtain the comprehensive loss of the fine-tuning stage. Among them, the weight coefficients of the cross entropy loss, modality contrast learning loss and balance loss function can be specifically configured and adjusted according to the needs of the actual application scenario, and are not specifically limited here. Furthermore, according to the comprehensive loss of the pre-training stage, the parameters of the video topic segmentation model to be trained are adjusted by back propagation.

[0182] In an optional embodiment, a Contrastive Semantic Similarity Learning (CSSL) task is introduced in the fine-tuning stage to achieve multimodal feature alignment between topics, so that the model pays more attention to enhancing the relevance of multimodal representations, and enables the model to identify relative consistency relationships within and between topics, which can help the model better capture the relationships and interactions between different topics, thereby improving the performance and accuracy of the model.

[0183] The fine-tuning stage also includes: calculating the semantic similarity contrast learning loss based on the multimodal fusion features of each sample fragment, and the semantic similarity contrast learning loss is used to increase the similarity of the multimodal fusion features of different sample fragments belonging to the same topic, and reduce the similarity of the multimodal fusion features of sample fragments belonging to different topics.

[0184] For example, in the fine-tuning stage, the following formula (3) can be used to calculate the semantic similarity contrast learning loss based on the multimodal fusion features of each sample segment:

[0185]

[0186] Among them, l mcssl Represents the semantic similarity contrast learning loss. n represents the number of data segments into which the video sample is segmented. m i represents the multimodal fusion feature of the i-th data segment. k1 represents the number of positive samples, k2 represents the number of negative samples, and k1 and k2 can be configured and adjusted according to the needs of the actual application scenario, and are not specifically limited here. Represents the multimodal fusion features of the jth positive sample of the i-th data segment. Represents the multimodal fusion features of the jth negative sample of the i-th data segment.

[0187] The positive sample of the i-th data segment refers to the data segment with the same theme as the i-th data segment, such as other data segments in the same video segment as the i-th data segment. The negative sample of the i-th data segment refers to the data segment that does not have the same theme as the i-th data segment, such as the data segment contained in the adjacent video segment of the video segment where the i-th data segment is located. By randomly extracting k1 positive samples from the positive samples of the i-th data segment and randomly extracting k2 negative samples from the negative samples of the i-th data segment, the semantic similarity contrast learning loss is calculated.

[0188] In addition, the calculation method of semantic similarity contrast learning loss can be modified based on formula (3), for example, Replace with Will Replace with Or it can be implemented by other cross entropy loss calculation methods, which are not specifically limited here.

[0189] Among them, the similarity of multimodal fusion features of different data fragments can be achieved by calculating the Euclidean distance, cosine similarity, or correlation coefficient between the multimodal fusion features of different data fragments, which is not specifically limited in this embodiment.

[0190] Furthermore, according to the semantic similarity contrast learning loss, the parameters of the video topic segmentation model to be trained are adjusted through back propagation.

[0191] For example, the cross entropy loss l′ can be calculated during the pre-training phase vts , modality contrastive learning loss l′ cma , semantic similarity contrastive learning loss l mcssl And the balance loss function l′ balance , for the cross entropy loss l′ vts , modality contrastive learning loss l′ cma , semantic similarity contrastive learning loss l mcssl And the balance loss function l′ balance The weighted summation gives the comprehensive loss in the fine-tuning stage, i.e., l finetune = l′ vts +σl′ balance +θl mcssl +γl′ cma Among them, σ, θ and γ are the balance loss function l′ balance , semantic similarity contrastive learning loss l mcssl and modality contrastive learning loss l′cma The weight coefficients σ, θ and γ are all greater than 0 and less than 1, which can be configured and adjusted according to the needs of the actual application scenario and are not specifically limited here. finetune , adjust the parameters of the video topic segmentation model to be trained through back propagation.

[0192] In this embodiment, during the fine-tuning stage, through cross-modal alignment and multimodal semantic similarity comparison learning tasks, the model pays more attention to enhancing the relevance of multimodal representations, so that the model enhances the consistency between visual features and text features, and can identify the relative consistency relationship within and between topics, thereby improving the quality of video topic segmentation results.

[0193] After the fine-tuning stage is completed, a trained video topic segmentation model is obtained. The performance of the video topic segmentation model can be improved through training in the fine-tuning stage.

[0194] For example, Fig. 9 The training architecture diagram of the video topic segmentation model provided in the embodiment of the present application. In the pre-training and fine-tuning stages, the calculated loss functions are not exactly the same, but the processing of video samples is the same. Fig. 9 As shown, the video sample is divided into n data segments, including text segments of n data segments and video clips

[0195]

[0196] Text fragments of n data fragments The input text encoding module extracts and outputs the text features of each text segment through the text encoder. The output features of the text encoding module are projected to a unified dimension through the projection layer to obtain the text features of each data segment {t1,…,t n}.

[0197] Video segments from n data segments The visual coding module is used to encode the n video clips and extract the visual features of each video clip. The output features of the visual coding module are projected to a unified dimension through the projection layer to obtain the visual features of each data clip {v1,…,v n}.

[0198] Furthermore, the text features of n data segments {t1,…,t n} and visual features {v1,…,v n} Input multimodal fusion module (including M multimodal fusion layers stacked in sequence, such as Fig. 9The text features and visual features of the n data segments are fused through the multimodal fusion module to obtain the fused text features of the n data segments. and fusion of visual features

[0199] The fused text features and fused visual features of each data segment are spliced ​​together to obtain the multimodal fusion features of each data segment. The multimodal fusion features of n data segments are represented as {m1,…,m n}.

[0200] Furthermore, the multimodal fusion features of n data segments {m1,…,m n} Input the classification prediction module, and the classification prediction module predicts the probability that each data segment is a topic boundary, which is expressed as {p1,…,p n}, and obtain the topic boundary prediction result of the video sample, that is, {p1,…,p n}.

[0201] In the pre-training stage, the prediction results {p1,…,p n}, and the subject boundary of the video sample, calculate the cross entropy loss (such as Fig. 9 The l shown vts ) According to the fused text features and fused visual features of each sample segment of the video sample, the modality contrast learning loss (such as Fig. 9 The l shown cma The purpose of this modality contrast learning loss is to increase the fused text features and fused visual features of the same sample segment (such as Fig. 9 Middle cma The similarity between the fused text features and fused visual features of different sample segments (as shown by the solid arrow below) is reduced. Fig. 9 Middle cma The similarity between them is shown by the dotted arrow below.

[0202] In the fine-tuning stage, the prediction results of the subject boundary {p1,…,p n}, and the subject boundary of the video sample, calculate the cross entropy loss (such as Fig. 9 The l shown vts ) According to the fused text features and fused visual features of each sample segment of the video sample, the modality contrast learning loss (such as Fig. 9 The l shown cma In addition, during the fine-tuning phase, a semantic similarity contrastive learning loss is added (e.g. Fig. 9 The l shown mcsslThe purpose of semantic similarity contrast learning loss is to increase the multimodal fusion features of different sample fragments of the same topic (such as Fig. 9 Middle mcssl The solid arrow below shows the similarity between the two, reducing the multimodal fusion features of sample clips of different topics (such as Fig. 9 Middle mcssl The similarity between them is shown by the dotted arrow below.

[0203] In addition, when the multimodal fusion module of the video topic segmentation model includes an attention layer and a mixed expert layer, the balance loss of the mixed expert layer can also be calculated in the pre-training stage and the fine-tuning stage, which is not shown in the figure.

[0204] In this embodiment, an initial video topic segmentation model is constructed based on a pre-trained visual encoder and a text encoder, and the initial video topic segmentation model is continuously pre-trained to obtain a pre-trained video topic segmentation model. In the pre-training (i.e., continuous pre-training) stage, the use of cross-modal alignment tasks and downstream task-related topic segmentation tasks helps to improve the model's results after fine-tuning. In the fine-tuning stage, the same cross-modal alignment tasks and downstream task-related topic segmentation tasks as in the pre-training stage are used. This continuity can ensure that the modality remains consistent during pre-training and fine-tuning, thereby promoting consistent interactions between different modalities. In addition, a multimodal semantic similarity comparison learning task is added in the fine-tuning stage. Through the cross-modal alignment tasks and multimodal semantic similarity comparison learning tasks in the fine-tuning stage, the model enhances the consistency between visual features and text features, and can identify the relative consistency relationship within and between topics, can accurately capture and utilize complementary information between different modalities, improve the overall consistency of information, and thus improve the quality of video topic segmentation results.

[0205] The video subject segmentation model trained by the method of this embodiment can perform better in scenes where complex and fuzzy subject boundaries are identified, can more accurately identify subject boundaries, and improve the accuracy and quality of video subject segmentation.

[0206] The method of the present application can be applied to application scenarios such as online education systems and video conference assistants. When implementing corresponding system functions, the video data in the system can be segmented into video topics. The following is an example of applying the method to a video conference assistant scenario to segment the video topic of a conference video.

[0207] Fig.10 Flow chart of a video segmentation method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. Fig.10 As shown, the specific steps of this method are as follows:

[0208] Step S1001: Obtain the recorded conference video.

[0209] When applied to video conference assistant scenarios, recording conference videos usually involves multiple different topics, such as different conference topics. By subject segmentation of conference videos, conference videos can be segmented into multiple video segments according to different conference topics. Adjacent video segments correspond to different conference topics, which helps relevant personnel quickly locate and access video segments corresponding to topics of interest.

[0210] After the video conference is over, the video conference assistant can send the recorded conference video to the server. The server receives the conference video sent by the terminal device where the video conference assistant is located.

[0211] For example, after the video conference assistant completes recording, it sends the video conference to the server in real time, triggering the server to segment the video conference. Alternatively, the video conference assistant can batch transmit the video conference recorded in this time period to the server at intervals. The server receives a batch of video conferences and segments each of them into video topics.

[0212] Step S1002: divide the conference video into multiple data segments, where the data segments include text segments and video segments.

[0213] Step S1003: input multiple data segments into the video topic segmentation model, extract the text features and visual features of each data segment through the video topic segmentation model, and predict whether each data segment is a topic boundary based on the text features and visual features of each data segment, so as to obtain the topic boundary of the conference video.

[0214] Step S1004: divide the video data into multiple video segments according to the subject boundaries of the conference video, wherein adjacent video segments correspond to different subjects.

[0215] In this embodiment, the implementation principle of steps S1002-S1004 for segmenting the conference video into video topics is specifically referred to the relevant contents of steps S201-S203 in the aforementioned embodiment, and this embodiment does not make any specific limitation here.

[0216] Step S1005: output information of multiple video segments of the conference video.

[0217] The server can also generate information about the multiple video segments into which the conference video is divided, including but not limited to: generating a theme and / or content summary corresponding to each video segment, determining the timestamp of each video segment (including a start timestamp and / or an end timestamp), and determining the start frame of each video segment.

[0218] Furthermore, the server can output information of the multiple video segments into which the conference video is divided through the front-end interface, wherein the information of the video segment includes at least one of the following: timestamp, starting video frame, subject, and content summary.

[0219] Users can select the video segments they are interested in based on the subject, content summary and other information of each video segment, and quickly find the video segment they are interested in and play it by fast forwarding, rewinding, dragging the progress bar and other methods based on the timestamp and other information of the video segment they are interested in.

[0220] Optionally, the user can select the information of the video segment that he is interested in, such as clicking on the conference topic of interest to select the video segment corresponding to the conference topic to be jumped to and play. In response to the selection operation of any video segment of the conference video, the conference video starts playing from the selected video segment.

[0221] The method of this embodiment divides the conference video into multiple data segments, including text segments and video segments, and extracts the features of a single modality of each data segment through a video topic segmentation model, including text features of the text modality and visual features of the visual modality, and then fuses the features of multiple modalities. It can fuse multimodal information at the intermediate representation level, better capture the relationship and interaction between different modalities, and obtain higher quality multimodal fusion features of each data segment. Furthermore, based on the multimodal fusion features of each data segment, it is possible to accurately predict whether each data segment is a topic boundary, obtain the topic boundary of the conference video, and improve the accuracy of topic boundary identification. Furthermore, based on the topic boundary of the conference video, the video data is divided into multiple video segments, which improves the accuracy and quality of the video topic segmentation results.

[0222] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present application. Fig.11 As shown, the server includes: a memory 1101 and a processor 1102. The memory 1101 is used to store computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1102 is connected to the memory 1101 in communication and is used to execute the computer-executable instructions stored in the memory 1101 to implement the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are similar and will not be repeated here.

[0223] Optional, such as Fig.11 As shown, the server also includes: a firewall 1103, a load balancer 1104, a communication component 1105, a power supply component 1106 and other components. Fig.11 Only some components are shown schematically, which does not mean that the server only includes Fig.11Components shown. Fig.11 In the description, the server is only taken as a cloud server deployed in the cloud as an example for exemplary description. The server can also be deployed locally, and this embodiment is not specifically limited here.

[0224] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method of any of the aforementioned embodiments is implemented. The specific functions and technical effects that can be achieved are not repeated here.

[0225] The present application also provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server executes the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0226] The embodiment of the present application provides a chip, including: a processing module and a communication interface, the processing module can execute the technical solution of the server in the aforementioned method embodiment. Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution provided by any of the aforementioned method embodiments.

[0227] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0228] It should be understood that the above processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in at least one processor.

[0229] The memory may include high-speed random access memory (Random Access Memory, RAM for short), and may also include non-volatile storage, such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc. The above-mentioned memory may be an object storage service (Object Storage Service, OSS for short). The above-mentioned memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, SRAM for short), electrically erasable programmable read-only memory (EEPROM for short), erasable programmable read-only memory (EPROM for short), programmable read-only memory (PROM for short), read-only memory (ROM for short), magnetic storage, flash memory, disk or optical disk.

[0230] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second-generation mobile communication system (2G), a third-generation mobile communication system (3G), a fourth-generation mobile communication system (4G) / Long Term Evolution (LongTerm Evolution, referred to as LTE), a fifth-generation mobile communication system (5G) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (Near Field Communication, referred to as NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (Radio Frequency Identification, referred to as RFID) technology, infrared technology, ultra-wideband (Ultra Wide Band, referred to as UWB) technology, Bluetooth technology and other technologies.

[0231] The above-mentioned power supply component provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing and distributing power for the device where the power supply component is located. The above-mentioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0232] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in a dedicated integrated circuit. Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0233] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0234] The order of the above-mentioned embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. In addition, in some of the processes described in the above-mentioned embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed or executed in parallel in the order in which they appear in this article, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0235] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0236] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0237] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video segmentation method, It is characterized in that include: Cutting the video data to be segmented into a plurality of data segments, wherein the data segments include text segments and video segments; Inputting the multiple data segments into a video topic segmentation model, extracting text features and visual features of each of the data segments through the video topic segmentation model, fusing the text features and visual features of each of the data segments to obtain multimodal fusion features of each of the data segments, and predicting whether each of the data segments is a topic boundary based on the multimodal fusion features of each of the data segments to obtain the topic boundary of the video data; The video data is divided into a plurality of video segments according to the subject boundaries of the video data, wherein adjacent video segments correspond to different subjects.

2. The method according to claim 1, It is characterized in that The step of dividing the video data to be segmented into a plurality of data segments comprises: Convert the audio of the video data to be segmented into text, and obtain text information corresponding to the audio; The text information is divided into a plurality of text segments, and the video data is segmented according to the timestamps of the plurality of text segments to obtain video segments corresponding to the text segments; Each of the text segments and the corresponding video segments constitute a data segment.

3. The method according to claim 1, It is characterized in that The video topic segmentation model includes: a text encoding module, a visual encoding module, a multimodal fusion module and a classification prediction module; The text features and visual features of each data segment are extracted by the video topic segmentation model, and the text features and visual features of each data segment are fused to obtain multimodal fusion features of each data segment, and according to the multimodal fusion features of each data segment, whether each data segment is a topic boundary is predicted to obtain a topic boundary prediction result, including: Extracting text features of text segments in each of the data segments through the text encoding module; Extracting visual features of the video segments in each of the data segments through the visual encoding module; By means of the multimodal fusion module, the text features and the visual features of each of the data segments are fused to obtain fused text features and fused visual features of each of the data segments; splicing the fused text features and fused visual features of each of the data segments to obtain multimodal fused features of each of the data segments; Through the classification prediction module, based on the multimodal fusion features of each of the data segments, classification prediction is performed on whether each of the data segments is a subject boundary, so as to obtain the subject boundary of the video data.

4. The method according to claim 3, It is characterized in that The multimodal fusion module includes an attention layer and a mapping layer. By means of the multimodal fusion module, the text features and visual features of the plurality of data segments are fused to obtain fused text features and fused visual features of each of the data segments, including: Performing attention calculation on the text features and visual features of the multiple data segments through the attention layer to obtain a multimodal fusion representation of each of the data segments; Through the mapping layer, the multimodal fusion representation of each data segment is mapped into fused text features and fused visual features.

5. The method according to claim 3, It is characterized in that The multimodal fusion module includes an attention layer and a mixed expert layer. By means of the multimodal fusion module, the text features and visual features of the plurality of data segments are fused to obtain fused text features and fused visual features of each of the data segments, including: By using the attention layer, the text features and visual features of the multiple data segments are subjected to attention calculation to obtain a multimodal fusion representation of each of the data segments; Through the hybrid expert layer, based on the hybrid expert mechanism, at least one expert module is selected for use to map the multimodal fusion representation of each of the data segments into fused text features and fused visual features.

6. The method according to claim 5, It is characterized in that The hybrid expert layer includes: a gating module, a merging module and a plurality of expert modules; Through the hybrid expert layer, based on the hybrid expert mechanism, at least one expert module is selected to map the multimodal fusion representation of each data segment into fused text features and fused visual features, including: Selecting at least one expert module according to the multimodal fusion representation of each of the input data segments through the gating module; By means of the at least one expert module, the multimodal fusion representation of each of the data segments is respectively mapped into fusion text features and fusion visual features of each of the data segments; The outputs of the at least one expert module are merged through the merging module according to the weight of the at least one expert module to obtain fused visual features of each data segment.

7. The method according to any one of claims 1 to 6, It is characterized in that Also includes: Outputting information of the plurality of video segments, wherein the information of the video segments includes at least one of the following: a timestamp, a starting video frame, a subject, and a content summary; In response to a selection operation on any of the video segments, the video data starts to be played from the selected video segment.

8. The method according to any one of claims 1 to 6, It is characterized in that The training process of the video topic segmentation model includes: In the pre-training stage, video samples with unknown subject boundaries are used to construct video samples with pseudo-annotated subject boundaries, and the video samples with pseudo-annotated subject boundaries are used to train the initial video subject segmentation model to obtain a pre-trained video subject segmentation model; In the fine-tuning stage, the pre-trained video topic segmentation model is trained using video samples with marked topic boundaries to obtain a trained video topic segmentation model.

9. The method according to claim 8, It is characterized in that In the pre-training stage, video samples with unknown subject boundaries are used to construct video samples with pseudo-annotated subject boundaries, including: Split a video sample whose subject boundary is unknown into multiple segments; For any video sample with unknown subject boundary, at least one segmented segment that does not belong to the video sample is inserted into the video sample, and / or at least one segmented segment of the video sample is replaced with a segmented segment that does not belong to the video sample, a new video sample is constructed, and the pseudo-annotated subject boundary of the new video sample is determined.

10. The method according to claim 9, It is characterized in that The step of dividing a video sample with unknown subject boundaries into a plurality of segmented segments includes: Get the duration distribution of different topics; For any video sample with unknown subject boundary, a duration is sampled according to the duration distribution, and the video sample is divided into a plurality of segmented segments according to the sampled duration.

11. The method according to claim 9, It is characterized in that In the pre-training and fine-tuning stages, video samples with topic boundaries are used to train the video topic segmentation model to be trained, including: Cutting a video sample with a subject boundary into a plurality of sample segments, wherein the sample segments include text segments and video segments; Inputting multiple sample segments of the video sample into a video topic segmentation model to be trained, extracting text features and visual features of each sample segment through the video topic segmentation model to be trained, fusing the text features and visual features of each sample segment to obtain multimodal fusion features of each sample segment, and predicting whether each sample segment is a topic boundary based on the multimodal fusion features of each sample segment, thereby obtaining a topic boundary prediction result of the video sample with a topic boundary; Calculating a cross entropy loss based on a prediction result of a topic boundary of the video sample and a topic boundary of the video sample; According to the cross entropy loss, the parameters of the video topic segmentation model to be trained are adjusted.

12. The method according to claim 11, It is characterized in that The text features and visual features of each sample segment are fused to obtain multimodal fusion features of each sample segment, including: Fusing the text features and visual features of the multiple data segments to obtain fused text features and fused visual features of each of the data segments; The fused text features and fused visual features of each of the data segments are spliced ​​to obtain multimodal fusion features of each of the data segments.

13. The method according to claim 12, It is characterized in that The pre-training stage and the fine-tuning stage further include: Calculating a modality contrastive learning loss based on the fused text features and the fused visual features of each of the sample segments of the video sample, wherein the modality contrastive learning loss is used to increase the similarity between the fused text features and the fused visual features of the same sample segment, and to reduce the similarity between the fused text features and the fused visual features of different sample segments; According to the modality contrast learning loss, parameters of the video topic segmentation model to be trained are adjusted.

14. The method according to claim 11, It is characterized in that The fine-tuning stage also includes: Calculating a semantic similarity contrast learning loss according to the multimodal fusion features of each of the sample segments, wherein the semantic similarity contrast learning loss is used to increase the similarity of the multimodal fusion features of different sample segments belonging to the same topic, and to reduce the similarity of the multimodal fusion features of sample segments belonging to different topics; According to the semantic similarity contrast learning loss, the parameters of the video topic segmentation model to be trained are adjusted.

15. The method according to claim 11, It is characterized in that The multimodal fusion module of the video topic segmentation model includes an attention layer and a mixed expert layer, and the pre-training stage and the fine-tuning stage also include: Calculating the balance loss of the hybrid expert layer; According to the balance loss, parameters of the hybrid expert layer are adjusted.

16. A video segmentation method, It is characterized in that include: Get recorded meeting videos; Cutting the conference video into multiple data segments, wherein the data segments include text segments and video segments; Input the multiple data segments into a video topic segmentation model, extract text features and visual features of each of the data segments through the video topic segmentation model, and predict whether each of the data segments is a topic boundary based on the text features and visual features of each of the data segments, so as to obtain the topic boundary of the conference video; According to the subject boundary of the conference video, the video data is divided into a plurality of video segments, wherein adjacent video segments correspond to different subjects; Output information of the multiple video segments of the conference video.

17. A server, It is characterized in that include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the server to execute the method described in any one of claims 1-16.

18. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 16 is implemented.

19. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Citation Information

Cited By

  • Image processing model training method and image processing method

    CN120746925A

  • Emotion detection method and device

    CN120877351A

  • Multi-modal question and answer method and device, electronic equipment, storage medium and program product

    CN121278147A

  • Multimodal question answering method and apparatus, electronic device, storage medium and program product

    CN121278147B

  • Multi-modal video sequence segmentation method based on text-guided hybrid expert mechanism

    CN121725393A