Video segmentation method, server, storage medium, and program product

Abstract

The present application provides a video segmentation method, a server, a storage medium, and a program product. In the method of the present application, video data to be segmented is segmented into multiple data segments, unimodal features of the data segments, including text features of a text modality and visual features of a visual modality, are respectively extracted by means of a video topics segmentation model, and then the text features and visual features of the data segments are fused, so that the fusion of multimodal information can be performed at the intermediate representation level, the relationship and interaction between different modalities can be better captured, and higher-quality multimodal fusion features of the data segments are obtained. Furthermore, on the basis of the multimodal fusion features of the data segments, whether the data segments are topic boundaries is predicted, so that the topic boundaries of the video data can be accurately predicted, improving the accuracy of topic boundary recognition, thereby improving the accuracy and quality of video topics segmentation results.

Description

Video segmentation method, server, storage medium and program product

[0001] The present disclosure claims priority to Chinese Patent Application No. 202410969075.4, filed on July 18, 2024, with the Chinese Patent Office, entitled "Video segmentation method, server, storage medium and program product", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of computer, and particularly relates to a video segmentation method, a server, a storage medium and a program product. BACKGROUND

[0003] In recent years, the proliferation of digital video content highlights the importance of efficient content navigation and understanding. Due to the unstructured nature of videos, it poses a significant challenge for users seeking to quickly grasp or reference specific topics. Therefore, video topic segmentation (VTS) has become an important tool to meet these needs. By dividing a video into coherent non-overlapping topic segments, VTS not only helps to intuitively understand the video content, but also enables quick positioning and access to the video segments of interest. This is particularly important for further carrying out various video understanding tasks.

[0004] At present, the video topic segmentation method based on clustering algorithm inputs the learned visual features and text features of each segment into a clustering algorithm to obtain the category to which each segment belongs. However, the effect of the clustering process is largely dependent on the selected algorithm and related parameter configuration. In addition, clustering may be sensitive to input features, and it is difficult to handle overlapping or fuzzy topic boundaries. In particular, in the case of smooth transition between topics or high correlation between topics, the clustering algorithm may have difficulty in accurately distinguishing different topics. Therefore, the video topic segmentation method has the problem of inaccurate topic boundary recognition, and the accuracy of the video topic segmentation result is low and the quality is poor. SUMMARY

[0005] The present disclosure provides a video segmentation method, a server, a storage medium and a program product to solve the problem of inaccurate topic boundary recognition, and the problem of low accuracy and poor quality of the video topic segmentation result.

[0006] In a first aspect, the present disclosure provides a video segmentation method, comprising: dividing video data to be segmented into a plurality of data segments, the data segments comprising text segments and video segments; inputting the plurality of data segments into a video theme segmentation model, extracting text features and visual features of each of the data segments by the video theme segmentation model, fusing the text features and the visual features of each of the data segments to obtain multi-modal fusion features of each of the data segments, and predicting whether each of the data segments is a theme boundary according to the multi-modal fusion features of each of the data segments to obtain theme boundaries of the video data; and segmenting the video data into a plurality of video segments according to the theme boundaries of the video data, wherein adjacent video segments correspond to different themes.

[0007] In a second aspect, the present disclosure provides a video segmentation method, comprising: obtaining a recorded conference video; dividing the conference video into a plurality of data segments, the data segments comprising text segments and video segments; inputting the plurality of data segments into a video theme segmentation model, extracting text features and visual features of each of the data segments by the video theme segmentation model, and predicting whether each of the data segments is a theme boundary according to the text features and the visual features of each of the data segments to obtain theme boundaries of the conference video; segmenting the video data into a plurality of video segments according to the theme boundaries of the conference video, wherein adjacent video segments correspond to different themes; and outputting information of the plurality of video segments of the conference video.

[0008] In a third aspect, the present disclosure provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the server to perform the method provided in any one of the preceding aspects.

[0009] In a fourth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method provided in any one of the preceding aspects is implemented.

[0010] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, and when a processor executes the computer program, the method provided in any one of the preceding aspects is implemented.

[0011] The video segmentation method, server, storage medium and program product provided by the present disclosure can cut the video data to be segmented into multiple data segments, extract the features of a single mode of each data segment through a video topic segmentation model, including text features of a text mode and visual features of a visual mode, fuse the text features and visual features of each data segment, can fuse the multi-modal information at the intermediate representation level, can better capture the relationship and interaction between different modes, and obtain higher quality multi-modal fusion features of each data segment; further, according to the multi-modal fusion features of each data segment, whether each data segment is a topic boundary is predicted, the topic boundary of the video data can be accurately predicted, the accuracy of topic boundary identification is improved, and thus the accuracy and quality of the video topic segmentation result are improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.

[0013] FIG. 1 is a schematic diagram of an example system architecture to which the present disclosure is applicable;

[0014] FIG. 2 is a flowchart of a video segmentation method according to an example embodiment of the present disclosure;

[0015] FIG. 3 is an architecture diagram of a video topic segmentation model according to an example embodiment of the present disclosure;

[0016] FIG. 4 is an example architecture diagram of a multi-modal fusion layer according to an example embodiment of the present disclosure;

[0017] FIG. 5 is another example architecture diagram of a multi-modal fusion layer according to an example embodiment of the present disclosure;

[0018] FIG. 6 is an example architecture diagram of a multi-modal fusion layer including a MoE layer according to an example embodiment of the present disclosure;

[0019] FIG. 7 is an example architecture diagram of another multi-modal fusion layer including a MoE layer according to an example embodiment of the present disclosure;

[0020] FIG. 8 is a flowchart of a training method of a video topic segmentation model according to an example embodiment of the present disclosure;

[0021] FIG. 9 is a training architecture diagram of a video topic segmentation model according to an example embodiment of the present disclosure;

[0022] FIG. 10 is a flowchart of a video segmentation method according to an example embodiment of the present disclosure;

[0023] FIG. 11 is a structural schematic diagram of a server according to an example embodiment of the present disclosure.

[0024] The specific embodiments of the present disclosure have been shown by the above-described drawings, and will be described in more detail hereinafter. These drawings and the written description are not intended to restrict the scope of the concept of the present disclosure in any way, but to illustrate the concept of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0025] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, the same numbers refer to the same elements throughout. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0026] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or rejection.

[0027] First, the terms involved in the present disclosure are explained:

[0028] Text Topic Segmentation (TTS): dividing a text sequence into a series of topic non-overlapping and consistent within the segment.

[0029] Video Topic Segmentation (VTS): dividing a video into a series of topic non-overlapping and consistent within the segment.

[0030] Multimodal Fusion (MF): modeling visual, text and other multi-modal input data at the same time.

[0031] Coherence Modeling (CM): a process that ensures logical and structural consistency between different data modes or characteristics in data analysis and processing.

[0032] Contrastive Learning (CL): learning the feature representation of a sample by comparing the sample with positive and negative samples in the feature space.

[0033] Pretraining: Learning general language representations from large-scale corpus, providing general model initialization method for various downstream tasks.

[0034] Continue Pretraining: Based on pre-training model, continue pre-training process on unlabeled data related to downstream tasks, providing better model initialization method for downstream tasks.

[0035] Finetuning: Model tuning based on existing pre-training model on in-domain data.

[0036] Visual Question Answering Task: Determine the answer to the question from the visual information of the input image based on the input image and question.

[0037] Image Description Task: Generate description text of input image.

[0038] Visual Entailment Task: Predict the semantic relevance between input image and text, i.e. entailment, neutral or contradiction.

[0039] Point Representation and Understanding Task: Locate the image region corresponding to the input text in the input image based on the input text.

[0040] Image Generation Task: Generate image based on input description text.

[0041] Text-based Sentiment Classification Task: Predict sentiment classification information of input text.

[0042] Text Summarization Task: Generate summary information of input text.

[0043] Multimodal Task: Refers to downstream tasks involving multiple modal data such as image and text, such as visual question answering task, image description task, visual entailment task, point representation and understanding task, image generation task, etc.

[0044] Multimodal Pre-training Model: Refers to pre-training model involving multiple modal data such as image and text, which can be applied to multimodal task processing after fine-tuning training.

[0045] Pre-training language model: Pre-training model obtained by pre-training large language model (LLM).

[0046] A large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, billions, or even tens of billions of model parameters. A large model can also be referred to as a foundation model (FM). Through large-scale unlabeled corpus pre-training, a pre-trained model with hundreds of millions of parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLM) and multi-modal pre-training models.

[0047] In practical applications, a large model only needs a small amount of sample data to fine-tune the pre-trained model and can be applied to different tasks. Large models can be widely used in natural language processing (NLP) and computer vision fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, and others.

[0048] To address the problem of inaccurate theme boundary identification in existing video theme segmentation methods, and the problem of low precision and poor quality of video theme segmentation results, the present disclosure provides a video segmentation method. The video data to be segmented is divided into multiple data segments, including text segments and video segments. The multiple data segments are input into a video theme segmentation model. The video theme segmentation model extracts text features and visual features of each data segment, fuses the text features and visual features of each data segment, obtains multi-modal fusion features of each data segment, and predicts whether each data segment is a theme boundary based on the multi-modal fusion features of each data segment to obtain the theme boundary of the video data. The video data is segmented into multiple video segments based on the theme boundary of the video data, wherein adjacent video segments correspond to different themes.

[0049] The video topic segmentation model is pre-constructed and is a machine learning model pre-trained and fine-tuned. The video topic segmentation model comprises a text encoding module, a visual encoding module, a multi-modal fusion module, and a classification prediction module. The text encoding module is configured to extract text features of the text segments of the data segments. The visual encoding module is configured to extract visual features of the video segments of the data segments. The multi-modal fusion module is configured to fuse the text features and the visual features of the data segments to obtain fused text features and fused visual features of the data segments. The fused text features and the fused visual features of the data segments are spliced to obtain multi-modal fusion features of the data segments, which are input into the classification prediction module. The classification prediction module is configured to classify and predict whether the data segments are topic boundaries according to the multi-modal fusion features of the data segments to obtain the topic boundaries of the video data.

[0050] The method of the embodiment extracts single modal features of the data segments, including text features of the text modal and visual features of the visual modal, through the video topic segmentation model, and then fuses the features of multiple modalities. The multi-modal information can be fused at the intermediate representation level, the relationship and interaction between different modalities can be better captured, and higher quality multi-modal fusion features of the data segments can be obtained. Further, whether the data segments are topic boundaries can be accurately predicted according to the multi-modal fusion features of the data segments, the accuracy of topic boundary recognition is improved, and thus the accuracy and quality of the video topic segmentation result are improved.

[0051] In the embodiment of the disclosure, the video topic segmentation task is taken as a segment-level sequence labeling task. The video data is divided into a plurality of data segments, the plurality of data segments are arranged in sequence to form a data segment sequence, and the data segments serving as topic boundaries are predicted, that is, each data segment is labeled as a topic boundary or not a topic boundary. Thus, it can be determined which data segments are topic boundaries.

[0052] The topic boundary here is the last data segment of any topic. After the last data segment of the multiple topics is known, the video data is divided from the end of the data segment serving as the topic boundary, and multiple video segments are obtained. Each video segment comprises one or more data segments, and the data segments in each video segment have the same topic. Adjacent video segments have different topics.

[0053] In addition, the topic boundary can also be defined as the first data segment of any topic. After the first data segment of the multiple topics is known, the video data is divided from the beginning of the data segment serving as the topic boundary, and multiple video segments are obtained. Each video segment comprises one or more data segments, and the data segments in each video segment have the same topic. Adjacent video segments have different topics.

[0054] FIG. 1 is a schematic diagram of an example system architecture to which the present disclosure is applicable. As shown in FIG. 1, the system architecture includes a server and an end-side device. The server and the end-side device have a communicable communication link therebetween, enabling a communication connection between the server and the end-side device.

[0055] The server is a computing-capable device deployed in the cloud or locally, such as a cloud cluster. The server stores a video topic segmentation model, which is used to classify and predict whether each data segment is a topic boundary according to a plurality of data segments (including text segments and video segments) divided from video data, to obtain a topic boundary prediction result. The server divides the video data to be segmented into a plurality of data segments based on the video data provided by the end-side device, predicts whether each data segment is a topic boundary by the video topic segmentation model, and obtains the topic boundary of the video data. The video data is segmented into a plurality of video segments according to the topic boundary of the video data, and the topic segmentation of the video data is realized.

[0056] In addition, the server can also be responsible for the pre-training and fine-tuning of the video topic segmentation model to obtain a trained video topic segmentation model. Of course, the pre-training and fine-tuning of the video topic segmentation model can also be completed by other servers.

[0057] The end-side device can be an electronic device running a downstream application system, which can be a hardware device with network communication function, operation function and information display function, including but not limited to smart phones, tablet computers, desktop computers, local servers, cloud servers, etc. The end-side device needs to perform video topic segmentation when running the downstream application system. The downstream application system running on the end-side device can be an online education system, a video conference assistant, etc., which needs to perform video topic segmentation on the video data in the system when realizing the corresponding system functions.

[0058] For example, in an online education system, a lecture video involves multiple different topics, such as different knowledge points. By performing topic segmentation on the lecture video, the lecture video can be segmented into multiple video segments according to the different knowledge points, and the adjacent video segments correspond to different knowledge points, which helps users quickly locate and access the video segments corresponding to the knowledge points of interest.

[0059] For example, in a video conference assistant, a conference video involves multiple different topics, such as different conference topics. By performing topic segmentation on the conference video, the conference video can be segmented into multiple video segments according to the different conference topics, and the adjacent video segments correspond to different conference topics, which helps relevant personnel quickly locate and access the video segments corresponding to the topics of interest.

[0060] Based on the system architecture shown in FIG. 1, when video topic segmentation is needed, the end-side device sends the video data to be segmented to the server. The server splits the video data to be segmented into multiple data segments, the data segments including text segments and video segments; inputs the multiple data segments into a video topic segmentation model, extracts text features and visual features of each data segment through the video topic segmentation model, fuses the text features and the visual features of each data segment to obtain multi-modal fusion features of each data segment, and obtains a topic boundary of the video data; and segments the video data into multiple video segments according to the topic boundary of the video data, wherein adjacent video segments correspond to different topics.

[0061] In an example scenario, the server can return the topic segmentation result of the video data, i.e., the segmentation information of the multiple video segments of the video data, to the end-side device. The end-side device outputs the information of the multiple video segments of the video data through a front-end interface, the information of the video segments including at least one of the following: a timestamp, a starting video frame, a topic, and a content abstract. In response to a selection operation on any video segment, the end-side device starts playing the video data from the selected video segment.

[0062] In another example scenario, the server can output the information of the multiple video segments of the video data through a front-end interface, the information of the video segments including at least one of the following: a timestamp, a starting video frame, a topic, and a content abstract. In response to a selection operation on any video segment, the server starts playing the video data from the selected video segment.

[0063] The scheme of the embodiment can be applied to video topic segmentation in various fields, including but not limited to online education and application scenarios of video conference assistants.

[0064] The technical scheme of the present disclosure and how the technical scheme of the present disclosure solves the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments. The embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0065] FIG. 2 is a flowchart of a video segmentation method according to an example embodiment of the present disclosure. The execution subject of the embodiment is the server in the system architecture described above. As shown in FIG. 2, the method includes the following specific steps:

[0066] In step S201, the video data to be segmented is split into multiple data segments, the data segments including text segments and video segments.

[0067] The video data to be segmented can be video data generated by different application scenarios in various fields, such as lecture videos in an online education system and conference videos generated by a video conference assistant, and can be video data in various formats. The source and format of the video data are not limited in this embodiment.

[0068] In this embodiment, the video topic segmentation is taken as a sequence labeling task at the segment level, the video data to be segmented is divided into a plurality of segments (referred to as data segments), and whether each data segment is a topic boundary (such as the last segment of a topic or the first segment of a topic) is predicted to identify the data segment that is a topic boundary in the sequence of segments into which the video data is divided, that is, to determine the topic boundary in the sequence of segments into which the video data is divided, which is also referred to as the topic boundary of the video data. Further, the video data is segmented into a plurality of video segments according to the topic boundary of the video data, so that any video segment corresponds to a unique topic, and adjacent video segments correspond to different topics.

[0069] Generally, the video data includes audio and video. In this step, when the video data is divided into a plurality of data segments, the audio of the video data is converted into text to obtain text information corresponding to the audio. The text information is divided into a plurality of text segments.

[0070] Specifically, when the audio of the video data is subjected to automatic speech recognition (ASR) to obtain text information corresponding to the audio, the audio is input into an ASR system. The ASR system analyzes the audio and generates a plurality of candidate texts (usually including a plurality of sentences and time stamps of the sentences), and ranks the candidate texts from high to low according to probabilities. The candidate text with the highest probability (i.e., the "1-best") is the most accurate recognition result of the ASR system for the current input audio, which is taken as the text information corresponding to the audio. In addition, methods other than ASR can be used to convert the audio of the video data into corresponding text information, such as a speech-to-text method based on machine learning, which is not limited herein.

[0071] In the implementation of dividing the text information into a plurality of text segments, the text information can be divided at the sentence granularity to divide the text information into a plurality of sentences, each sentence being a text segment. Alternatively, when the text information is divided into a plurality of text segments, the text information can be divided at a granularity coarser than the sentence granularity (such as paragraph granularity or semantic block granularity) to divide the text information into a plurality of text segments at a coarser granularity.

[0072] In this step, rule-based text segmentation, statistical-based text segmentation and machine learning-based text segmentation methods can be used to divide the text information into multiple text segments, and the text segment division method is not limited in the embodiment.

[0073] Further, the video data is divided according to the timestamps of the multiple text segments to obtain video segments corresponding to the text segments. Each text segment and the corresponding video segment constitute a data segment. Each text segment obtained by division contains one or more sentences. The timestamp of the text segment (including the start time and the end time) can be determined according to the timestamp of each sentence contained in the text segment.

[0074] In addition, when the video data is divided into multiple data segments, the existing video topic segmentation scheme that uses video topic segmentation as a segment-level sequence labeling task can be used to divide the video data to be segmented. For example, according to a voice activity detection method, the voice of the video data is divided from the pause to obtain multiple voice segments, the language segments are automatically converted into corresponding text segments by a speech recognition device, and the corresponding video segments are determined according to the timestamp of the voice segment. Each voice segment, the corresponding text segment and the video segment constitute a data segment.

[0075] For example, in the disclosure, n represents the number of data segments obtained by dividing the video data, and the n data segments can be represented as wherein, represents the video segment contained in the i-th data segment, and the video segment includes k video frames can be represented as represents the text segment contained in the i-th data segment, and the text segment is a word sequence, which can be represented as wherein, ‖s i represents the number of words contained in the text segment contained in the i-th data segment. In addition, a token can be added at the start position (or the end position) of the text segment , which can be represented as represents a special mark inserted at the start position, such as “[BOS]”, which is used to identify the start position of a text segment. Alternatively, represents a special mark inserted at the end position, which is used to identify the end position of a text segment.

[0076] In step S202, the plurality of data segments are input into the video topic segmentation model, the text features and the visual features of each data segment are extracted by the video topic segmentation model, the text features and the visual features of each data segment are fused to obtain the multi-modal fusion features of each data segment, and whether each data segment is a topic boundary is predicted according to the multi-modal fusion features of each data segment to obtain the topic boundary of the video data.

[0077] The video topic segmentation model is a machine learning model that is pre-constructed and pre-trained and fine-tuned.

[0078] In this embodiment, the plurality of data segments obtained by cutting the video data are input into the video topic segmentation model, the features of the text segments of each data segment are extracted by the text encoding module of the video topic segmentation model to obtain the text features of each data segment, and the features of the video segments of each data segment are extracted by the visual encoding module to obtain the visual features of each data segment. The text features and the visual features of each data segment are fused by the multi-modal fusion module, which can fuse multi-modal information at the intermediate representation level, better capture the relationship and interaction between different modalities, and obtain the multi-modal fusion features of each data segment. Whether each data segment is a topic boundary is classified and predicted by the classification prediction module according to the multi-modal fusion features of each data segment. According to the classification prediction result, it can be determined which data segments are the topic boundary, so as to obtain the topic boundary of the video data.

[0079] In step S203, the video data is segmented into a plurality of video segments according to the topic boundary of the video data, wherein adjacent video segments correspond to different topics.

[0080] After obtaining the topic boundary of the video data, the video data is segmented from the topic boundary to obtain a plurality of video segments. Each video segment contains one or more data segments, and the data segments contained in the same video segment correspond to the same topic, that is, the same video segment corresponds to a unique topic. In addition, the plurality of video segments obtained by segmentation are arranged in chronological order of timestamps, and adjacent video segments correspond to different topics.

[0081] In an example scenario, after the video data is segmented into a plurality of video segments, the server can also generate information of the plurality of video segments obtained by segmenting the video data, including but not limited to: generating the topic and / or content summary corresponding to each video segment, determining the timestamp (including the start timestamp and / or the end timestamp) of each video segment, and determining the start frame of each video segment.

[0082] The server can output information of the plurality of video segments through a front-end interface. The information of the video segments includes at least one of a timestamp, a starting video frame, a theme, and a content abstract.

[0083] The user can select a video segment of interest according to the theme, the content abstract, or the like of each video segment, and quickly find the video segment of interest for playing according to the timestamp or the like of the video segment of interest through fast-forwarding, rewinding, dragging a progress bar, or the like.

[0084] Alternatively, the user can perform a selection operation on the information of the video segment of interest, such as clicking the information of the video segment of interest, to select the video segment for which to jump to play. In response to the selection operation on any video segment, the server starts playing the video data from the selected video segment, so as to automatically locate the video segment of interest of the user.

[0085] The method of the embodiment extracts single modal features of each data segment, including text features of a text modal and visual features of a visual modal, through a video theme segmentation model, and then fuses the features of multiple modals, so that the fusion of multi-modal information can be performed at an intermediate representation level, the relationship and interaction between different modals can be better captured, and higher-quality multi-modal fusion features of each data segment can be obtained. Further, whether each data segment is a theme boundary can be accurately predicted according to the multi-modal fusion features of each data segment, the accuracy of theme boundary identification is improved, and thus the accuracy and quality of the video theme segmentation result are improved.

[0086] FIG. 3 is an architecture diagram of a video theme segmentation model provided by an example embodiment of the present disclosure. As shown in FIG. 3, the video theme segmentation model includes a text encoding module, a visual encoding module, a multi-modal fusion module, and a classification prediction module.

[0087] The text encoding module is configured to extract text features of the text segment of each data segment. The text encoding module can use a pre-trained text encoder, such as a BERT (Bidirectional Encoder Representations from Transformers) encoder, a Transformer encoder, a pre-trained language model, or other encoders for encoding text information, which is not specifically limited herein.

[0088] The visual encoding module is configured to extract visual features of the video segments in the data segments. The visual encoding module can employ a pre-trained visual encoder, such as a Vision Transformer (ViT), a Convolutional Neural Network (CNN), a Long Short-Term Memory (LSTM), a Contrastive Language-Image Pre-training (CLIP) model, or the like, without limitation.

[0089] Optionally, to convert the features of different modalities into the same dimension, a projection layer can be added after the text encoder and the visual encoder, respectively, to project the text features output by the text encoder and the visual features output by the visual encoder into the same dimension, so that the dimensions of the text features and the visual features of the data segments after projection are consistent.

[0090] The multi-modal fusion module is configured to fuse the text features and the visual features of the data segments to obtain fused text features and fused visual features of the data segments.

[0091] The multi-modal fusion module can include a plurality of multi-modal fusion layers stacked in sequence (e.g., “Mx” in FIG. 3, indicating M multi-modal fusion layers stacked in sequence). Each multi-modal fusion layer is configured to fuse the text features and the visual features of the data segments input into the layer to output fused text features and fused visual features of the data segments. The fused text features and the fused visual features of the data segments output by the last multi-modal fusion layer are taken as the fused text features and the fused visual features of the data segments.

[0092] Further, the fused text features and the fused visual features of the data segments are spliced to obtain multi-modal fusion features of the data segments, which are input into the classification prediction module.

[0093] The classification prediction module is configured to classify and predict whether each data segment is a topic boundary based on the multi-modal fusion features of the data segments to obtain the topic boundaries of the video data. The classification prediction module can employ a binary classification predictor to predict the probability that each data segment is a topic boundary. The prediction result output by the classification prediction module includes the probability that each data segment is a topic boundary. Further, a data segment with a probability greater than or equal to a probability threshold that it is a topic boundary is taken as a topic boundary, while a data segment with a probability less than the probability threshold that it is a topic boundary is not taken as a topic boundary.

[0094] The probability threshold can be configured and adjusted according to the requirements of the actual application scene, for example, the probability threshold can be configured as 0.5 or other values, which is not limited here.

[0095] Based on the video topic segmentation model shown in FIG. 3, in the foregoing step S202, the text features and visual features of each data segment are extracted by the video topic segmentation model, the text features and visual features of each data segment are fused to obtain the multi-modal fusion features of each data segment, and whether each data segment is a topic boundary is predicted according to the multi-modal fusion features of each data segment to obtain a topic boundary prediction result. The foregoing steps are implemented in the following manner:

[0096] The text features of each text segment in the n data segments are extracted by the text encoding module. The visual features of each video segment in the n data segments are extracted by the visual encoding module. Further, the text features and visual features of the plurality of data segments are fused by the multi-modal fusion module to obtain the fused text features and fused visual features of each data segment. The fused text features and fused visual features of each data segment are spliced to obtain the multi-modal fusion features of each data segment. Further, whether each data segment is a topic boundary is classified and predicted according to the multi-modal fusion features of each data segment by the classification prediction module to obtain the topic boundary of the video data.

[0097] Exemplarily, as shown in FIG. 3, the text segments in the n data segments segmented from the video data are input into the text encoding module. The n text segments input into the text encoding module are encoded by the text encoding module, and the text features of each text segment are extracted and output. The output features of the text encoding module are projected to a unified dimension by a projection layer to obtain the text features of each data segment. The text features of the n data segments can be represented as: {t1, …, tn}. n}.

[0098] The video segments in the n data segments segmented from the video data are input into the visual encoding module. The n video segments input into the visual encoding module are encoded by the visual encoding module, and the visual features of each video segment are extracted. The output features of the visual encoding module are projected to a unified dimension by a projection layer to obtain the visual features of each data segment. The visual features of the n data segments can be represented as: {v1, …, vn}. n}.

[0099] Optionally, since the calculation complexity of the visual encoding module is high, the video segments in each data segment can be respectively input into the visual encoding module for encoding; or the n video segments are divided into a plurality of groups, each group containing a smaller number (compared to the total number n) of video segments, and a smaller number of video segments in each group are input into the visual encoding module for encoding at a time.

[0100] Furthermore, the text features {t1,…,t} of the n data segments n} and visual features {v1,…,v n The input is a multimodal fusion module (comprising M stacked multimodal fusion layers). The multimodal fusion module processes the text features {t1,…,t} of n data segments. n} and visual features {v1,…,v n The data fragments are fused to obtain the fused text features. and fusion of visual features

[0101] The fused text features and fused visual features of each data segment are concatenated to obtain the multimodal fusion features of each data segment. The multimodal fusion features of n data segments are represented as {m1,…,m n}

[0102] Furthermore, the multimodal fusion features {m1,…,m} of n data segments n The input is given to the classification prediction module, which predicts the probability that each data segment is a topic boundary, denoted as {p1,…,p}. n}

[0103] Furthermore, based on the probability {p1,…,p} that each data segment is a topic boundary, n For any data segment i, if the probability p that the data segment is a topic boundary is... i If the probability p is greater than or equal to the probability threshold, then the data segment is determined to be a topic boundary. i If the probability is less than the probability threshold, then the data segment is determined not to be a topic boundary.

[0104] The video topic segmentation model provided in this embodiment extracts textual and visual features from each data segment using text encoding and visual encoding modules, respectively, to obtain features of multiple single modalities. Then, a multimodal fusion module fuses these features, enabling multimodal information fusion at an intermediate representation level. This allows for better capture of the relationships and interactions between different modalities, resulting in higher-quality multimodal fusion features for each data segment. Furthermore, a classification prediction module, based on the multimodal fusion features of each data segment, can accurately predict whether each data segment represents a topic boundary, improving the accuracy of topic boundary recognition and thus enhancing the accuracy and quality of the video topic segmentation results.

[0105] In an optional embodiment, the multi-modal fusion module can include an attention layer and a mapping layer. The attention layer is configured to perform attention calculation on the text features and the visual features of the plurality of data segments to obtain multi-modal fusion representations of the data segments. The mapping layer is configured to map the multi-modal fusion representations of the data segments into fusion text features and fusion visual features.

[0106] The attention layer can be implemented by using a self-attention module or a neural network combining self-attention and cross-attention. The mapping layer can be implemented by using a feedforward neural network (FNN) or a multilayer perceptron (MLP).

[0107] For example, the multi-modal fusion module can be implemented based on a merge-attention mechanism. As shown in FIG. 4, the multi-modal fusion layer in the multi-modal fusion module can be implemented by using a merge-attention layer. The merge-attention layer includes a self-attention layer and a feedforward neural network layer. The merge results of the text features and the visual features of the plurality of data segments are input into the self-attention layer of the first layer of the merge-attention layer in the multi-modal fusion module to perform self-attention calculation, thereby obtaining multi-modal fusion representations of the plurality of data segments. The multi-modal fusion representations of the plurality of data segments are input into the feedforward neural network layer, and the feedforward neural network layer is configured to map the multi-modal fusion representations of the plurality of data segments into text features and visual features, thereby obtaining fusion text features and fusion visual features of the plurality of data segments. The fusion text features and the fusion visual features output by the last layer of the multi-modal fusion layer in the multi-modal fusion module are taken as the fusion text features and the fusion visual features finally output by the multi-modal fusion module.

[0108] For example, the input of a layer of the multi-modal fusion layer in the multi-modal fusion module includes fusion text features and fusion visual features output by the previous layer of the multi-modal fusion layer (denoted by l-1). The merge results of the input fusion text features and the input fusion visual features are The merge results are taken as query features (denoted by Q v；t in the figure), key features (denoted by K v；t in the figure), and value features (denoted by V v；t in the figure) respectively, and are input into the self-attention layer of the current layer of the multi-modal fusion layer (denoted by l) to perform self-attention calculation, thereby obtaining multi-modal fusion representations of the n data segments, which can be denoted as Further, the multi-modal fusion representations of the n data segments are mapped into text features by using the feedforward neural network layer. and visual features The fused text features and the fused visual features output by the current multi-modal fusion layer (denoted by l) are taken as the output of the multi-modal fusion module. and fused visual features

[0109] Exemplarily, the multi-modal fusion module can be implemented based on a Co-Attention mechanism. As shown in FIG. 5, the multi-modal fusion layers in the multi-modal fusion module can be implemented by Co-Attention layers. Each Co-Attention layer includes two branches, each of which includes a self-attention layer, a cross-attention layer, etc., and a feedforward neural network layer.

[0110] The text features of the plurality of data segments are taken as the input query features, the input key features and the input value features of the self-attention layer in one branch (denoted as the first branch), and are input into the self-attention layer in the first branch for self-attention calculation; the visual features of the plurality of data segments are taken as the input query features, the input key features and the input value features of the self-attention layer in the other branch (denoted as the second branch), and are input into the self-attention layer in the second branch for self-attention calculation. In FIG. 5, Q t , K t and V t represent the query features, the key features and the value features in the first branch (the branch corresponding to the text features), and Q v , K v and V v represent the query features, the key features and the value features in the second branch (the branch corresponding to the visual features).

[0111] Further, the output query features of the self-attention layer in the first branch are taken as the input query features of the cross-attention layer in the first branch, and the output key features and the output value features of the self-attention layer in the second branch are taken as the input key features and the input value features of the cross-attention layer in the first branch, which are input into the cross-attention layer in the first branch for cross-attention calculation, to obtain the multi-modal fusion representation of the first branch.

[0112] Further, the output query features of the self-attention layer in the first branch are taken as the input query features of the cross-attention layer in the first branch, and the output key features and the output value features of the self-attention layer in the second branch are taken as the input key features and the input value features of the cross-attention layer in the first branch, which are input into the cross-attention layer in the first branch for cross-attention calculation, to obtain the multi-modal fusion representation of the first branch.

[0113] Further, the multi-modal fusion representation of the first branch is input into a feedforward neural network layer of the first branch, and the multi-modal fusion representation of the first branch is mapped into text features by the feedforward neural network layer of the first branch, to obtain the fused text features of the plurality of data segments.

[0114] The multi-modal fusion representation of the second branch is input into a feedforward neural network layer of the second branch, and the multi-modal fusion representation of the second branch is mapped into visual features by the feedforward neural network layer of the second branch, to obtain the fused visual features of the plurality of data segments.

[0115] The fused text features and the fused visual features output by the last multi-modal fusion layer of the multi-modal fusion module are taken as the fused text features and the fused visual features finally output by the multi-modal fusion module.

[0116] In an optional embodiment, the multi-modal fusion module can include an attention layer and a mixture of experts (MoE) layer. The text features and the visual features of the plurality of data segments are subjected to attention calculation by the attention layer, to obtain the multi-modal fusion representation of each data segment. The multi-modal fusion representation of each data segment is mapped into the fused text features and the fused visual features by the mixture of experts layer based on the mixture of experts mechanism, by selecting to use at least one expert module.

[0117] Each multi-modal fusion layer in the multi-modal fusion module includes an attention layer and a mixture of experts. The attention layer can be implemented by using a self-attention module or by using a neural network combining self-attention and cross-attention. For example, the attention layer shown in FIG. 4 or FIG. 5. The mixture of experts layer includes a gating module, a merging module, and a plurality of expert modules.

[0118] The gating module selects at least one expert module according to the input multi-modal fusion representation of each data segment. The multi-modal fusion representation of each data segment is mapped into the fused text features and the fused visual features by the at least one expert module. The outputs of the at least one expert module are merged (weighted summation or weighted average) according to the weights of the at least one expert module by the merging module, to obtain the fused visual features of each data segment.

[0119] Each expert module is a mapping module, which can be implemented by using a feedforward neural network, a multilayer perceptron, etc.

[0120] The gating module can be a gating network based on a neural network (such as a convolutional layer, a fully connected layer), which is responsible for receiving input data and calculating the weight of each expert module according to the input data. These weights are usually normalized by a softmax function to ensure that the sum of the weights of all expert modules is 1, thereby forming a probability distribution. Alternatively, the gating module can also be implemented by other methods based on decision trees or random forests, etc., which are not specifically limited here.

[0121] Alternatively, in order to further improve the computational efficiency, a sparse gating mechanism can be introduced into the MoE layer, aiming to ensure that only a small part of the expert modules are activated for processing the input data. For example, a Top-k gating mechanism can be used to select the k expert modules with higher weights according to the weights of each expert module output by the gating module. This method can significantly reduce the amount of calculation while maintaining the expressiveness of the MoE module. For example, a threshold selection mechanism can be used to activate only the expert modules with weights greater than the weight threshold according to the weights of each expert module determined by the gating module.

[0122] The merging module is responsible for fusing the outputs of each expert module to obtain the final output result of the hybrid expert layer. The merging module ensures that the MoE module can integrate the capabilities of each expert module while maintaining the consistency and coherence of the overall output.

[0123] Exemplarily, the multi-modal fusion module is implemented by combining the hybrid expert mechanism and the merging attention mechanism. As shown in FIG. 6, the multi-modal fusion layer in the multi-modal fusion module includes a self-attention layer and a hybrid expert layer (i.e., an MoE layer). The merged results of the text features and the visual features of the plurality of data segments are input as query features, key features, and value features into the self-attention layer of the first layer of the merging attention layer in the multi-modal fusion module for self-attention calculation to obtain multi-modal fusion representations of the plurality of data segments; the multi-modal fusion representations of the plurality of data segments are input into the hybrid expert layer, and at least one expert module is selected by the hybrid expert layer based on the hybrid expert mechanism to map the multi-modal fusion representations of the data segments into fused text features and fused visual features. The fused text features and the fused visual features output by the last layer of the multi-modal fusion layer of the multi-modal fusion module are the fused text features and the fused visual features finally output by the multi-modal fusion module.

[0124] Exemplarily, FIG. 6 is a structural schematic diagram of a multi-modal fusion layer provided by the embodiment. The multi-modal fusion layer used in the embodiment can be realized by replacing the feedforward neural network layer in the multi-modal fusion layer shown in FIG. 4 with a mixed expert (i.e., MoE) layer. The implementation principle of the self-attention layer shown in FIG. 6 is consistent with that of the self-attention layer in FIG. 4. For details, refer to the related content of the foregoing embodiments, which will not be repeated here. As shown in FIG. 6, the multi-modal fusion representation of the n data segments output by the self-attention layer is input into the gating module of the MoE layer. The gating module selects at least one activated expert module from all s expert modules according to the input multi-modal fusion representation of each data segment, and provides the weight of each activated expert module to the merging module. Each activated expert module participates in the current calculation, and respectively maps the multi-modal fusion representation of each data segment into a fusion text feature and a fusion visual feature. The merging module weights and sums the output results of each expert module according to the weight of at least one expert module, to obtain the fusion text feature and the fusion visual feature of each data segment.

[0125] Exemplarily, FIG. 7 is a structural schematic diagram of another multi-modal fusion layer provided by the embodiment. The multi-modal fusion layer used in the embodiment can be realized by replacing the two feedforward neural network layers in the multi-modal fusion layer shown in FIG. 5 with one mixed expert (i.e., MoE) layer. The implementation principle of the co-attention layer shown in FIG. 7 is consistent with that of the co-attention layer in FIG. 5. For details, refer to the related content of the foregoing embodiments, which will not be repeated here. As shown in FIG. 7, the multi-modal fusion representation of the first branch and the multi-modal fusion representation of the second branch in the co-attention layer are input into the gating module of the MoE layer. The gating module selects at least one activated expert module according to the input multi-modal fusion representation of each data segment, and provides the weight of each activated expert module to the merging module. Each activated expert module participates in the current calculation, and respectively maps the multi-modal fusion representation of each data segment into a fusion text feature and a fusion visual feature. The merging module weights and sums the output results of each expert module according to the weight of at least one expert module, to obtain the fusion text feature and the fusion visual feature of each data segment.

[0126] In the embodiment, by introducing the MoE mechanism in the multi-modal fusion layer, the multi-modal fusion representation is input into the MoE layer for processing. The MoE layer can promote deeper cross-modal information fusion, can improve the model capacity without increasing the calculation complexity in proportion to the capacity, and can reduce the calculation complexity to a certain extent, thereby optimizing the multi-modal fusion process.

[0127] The video topic segmentation model in the foregoing embodiments can be obtained by constructing an initial video topic segmentation model and using a training set with labeled topic boundaries of a downstream task to supervise and train the video topic segmentation model.

[0128] In constructing the initial video topic segmentation model, a pre-trained neural network can be used. Specifically, the text encoding module and the visual encoding module can be implemented using a pre-trained text encoder and a pre-trained visual encoder, and the classification predictor can be implemented using a pre-trained classifier. The parameters of the multi-modal fusion module can be determined randomly.

[0129] To better initialize the parameters of the multi-modal fusion module in the video topic segmentation model, the initial video topic segmentation model can be pre-trained (here, continuous pre-training) using unlabeled video data before supervised training using a training set with labeled topic boundaries, and then fine-tuned using the training set with labeled topic boundaries.

[0130] FIG. 8 is a flowchart of a method for training a video topic segmentation model according to an example embodiment of the present disclosure. As shown in FIG. 8, the method can include the following steps:

[0131] At step S800, an initial video topic segmentation model is obtained.

[0132] In this embodiment, the architecture of the initial video topic segmentation model constructed is shown in FIG. 3, which includes a text encoding module, a visual encoding module, a multi-modal fusion module, and a classification prediction module. The parameters of the text encoding module can be initialized using the parameters of a pre-trained text encoder. The parameters of the visual encoding module can be initialized using the parameters of a pre-trained visual encoder. The parameters of the multi-modal fusion module are determined randomly. The parameters of the classification predictor can be initialized using the parameters of a pre-trained classifier, or can be initialized randomly.

[0133] In the pre-training phase and the fine-tuning phase, the parameters of the entire video topic segmentation model can be adjusted. In addition, to improve the training speed, the parameters of the visual encoding module and / or the text encoding module are frozen (fixed) in the pre-training phase and the fine-tuning phase, and the parameters of the remaining modules are adjusted. In a preferred embodiment, the parameters of the visual encoding module can be frozen, and the parameters of the text encoding module, the multi-modal fusion module, and the classification prediction module can be adjusted.

[0134] At step S801, in the pre-training phase, a video sample with unknown topic boundaries is used to construct a video sample with pseudo-labeled topic boundaries, and the initial video topic segmentation model is trained using the video sample with pseudo-labeled topic boundaries to obtain a pre-trained video topic segmentation model.

[0135] In this embodiment, in the pre-training phase, the initial video topic segmentation model is trained using unlabeled video samples to obtain a pre-trained video topic segmentation model.

[0136] Specifically, first, a video sample with unknown theme boundaries is used to construct a video sample with pseudo-labeled theme boundaries. Then, the video sample with pseudo-labeled theme boundaries is used to supervise the training of an initial video theme segmentation model to obtain a pre-trained video theme segmentation model.

[0137] In constructing the video sample with pseudo-labeled theme boundaries, the video sample with unknown theme boundaries is cut into multiple segments, each segment is called a cut segment; for any video sample with unknown theme boundaries, at least one cut segment not belonging to the video sample is inserted into the video sample, and / or at least one cut segment of the video sample is replaced with a cut segment not belonging to the video sample, a new video sample is constructed, and the pseudo-labeled theme boundaries of the new video sample are determined.

[0138] In the new video sample, the newly inserted cut segment has a different theme from the adjacent cut segment, and the replaced cut segment has a different theme from the adjacent cut segment. By discontinuously inserting (or replacing) cut segments of other video samples into the video sample, the theme of the inserted (or replaced) cut segment is different from that of the adjacent cut segment, thereby constructing a determined theme boundary.

[0139] For the constructed new video sample, the video sample is cut into multiple sample segments. For any two adjacent and different theme cut segments in the new video sample, the last sample segment in the former cut segment (or the first sample segment in the latter cut segment) is taken as a theme boundary. Thus, the pseudo-labeled theme boundary of the new video sample can be determined. Since the theme boundary is automatically generated, not obtained by labeling the video sample, the constructed theme boundary is referred to as a pseudo-labeled theme boundary.

[0140] Optionally, in implementing the cutting of the video sample with unknown theme boundaries into multiple cut segments, the duration distribution of the theme is obtained; for any video sample with unknown theme boundaries, a duration is sampled according to the duration distribution, and the video sample is cut into multiple cut segments according to the sampled duration.

[0141] The duration distribution of a topic can be obtained by counting and analyzing the duration of video segments of different topics in the video samples with annotated topic boundaries. For example, a Kernel Density Estimation (KDE) model or other statistical methods can be used to approximate the duration distribution of each topic in the video samples with annotated topic boundaries. For example, the duration range of a topic can be determined according to the minimum duration and the maximum duration of each topic in the video samples, and the duration range of the topic can be divided into multiple intervals as the duration intervals of the topic. According to the duration of the video segments of each topic in the video samples, the proportion of the number of video segments falling into each duration interval (as the sampling probability of the corresponding duration interval of the topic) is counted to obtain the duration distribution of the topic.

[0142] Specifically, when any video sample is divided into multiple divided segments, according to the duration distribution of the topic, the proportion of the number of video segments falling into each duration interval of the topic can be used as the sampling probability of each duration interval, a duration interval is sampled, a value is randomly sampled from the duration interval as the duration, and the video sample is divided into multiple divided segments according to the sampled duration.

[0143] Optionally, when a video sample with unknown topic boundaries is divided into multiple divided segments, a duration can also be randomly sampled from the pre-configured duration range of the topic, and the video sample is divided into multiple divided segments according to the sampled duration.

[0144] Further, when a new video sample is constructed, for any video sample with unknown topic boundaries, each divided segment in the video sample is operated with equal probability to one of the following three operations: inserting a divided segment randomly selected from other video samples before or after the current divided segment, replacing the current divided segment with a divided segment randomly selected from other video samples, and making no changes. In this way, at least one divided segment not belonging to the video sample can be inserted into the video sample, and / or at least one divided segment of the video sample can be replaced by a divided segment not belonging to the video sample, to construct a new video sample.

[0145] After obtaining the video samples with pseudo-annotated topic boundaries, in the pre-training stage, the initial video topic segmentation model is trained using the video samples with pseudo-annotated topic boundaries as follows:

[0146] The video sample with pseudo-labeled theme boundary is cut into multiple sample segments, and the sample segments include text segments and video segments; the multiple sample segments of the video sample are input into the video theme segmentation model to be trained, the text features and visual features of the sample segments are extracted by the video theme segmentation model to be trained, the text features and visual features of the sample segments are fused to obtain the multi-modal fusion features of the sample segments, and whether each sample segment is a theme boundary is predicted according to the multi-modal fusion features of the sample segments to obtain the theme boundary prediction result of the video sample.

[0147] According to the theme boundary prediction result of the video sample and the theme boundary of the video sample, the cross-entropy loss is calculated.

[0148] Exemplarily, in the case that the theme boundary is defined as the last data segment in the theme, the last data segment of the video sample does not calculate the loss. The cross-entropy loss is calculated by the following formula (1):

[0149] Wherein, l vts represents the cross-entropy loss, p i represents the probability that the i-th data segment is a boundary theme. y i represents the label of whether the i-th data segment is a theme boundary, y i = 1 indicates that the i-th data segment is a theme boundary, y i = 0 indicates that the i-th data segment is not a theme boundary. ln represents the natural logarithm. n represents the number of data segments.

[0150] In addition, the cross-entropy loss function can be deformed based on formula (1), for example, ln is replaced by log with base 10, or other cross-entropy loss calculation methods can be used to realize, which is not limited here.

[0151] Further, according to the cross-entropy loss, the parameters of the video theme segmentation model to be trained are adjusted by back propagation.

[0152] In an optional embodiment, in the pre-training stage, when the text features and visual features of the sample segments are fused by the multi-modal fusion module to obtain the multi-modal fusion features of the sample segments, the multi-modal fusion module first fuses the text features and visual features of the multiple data segments to obtain the fused text features and fused visual features of the data segments; then the fused text features and fused visual features of the data segments are spliced to obtain the multi-modal fusion features of the data segments.

[0153] In the pre-training stage, a cross-modal alignment task is introduced, and a contrastive learning loss is used to adjust the features learned by the multi-modal fusion module in the model, to maximize the similarity of the fused text features and the fused visual features of the same data segment, while reducing the similarity of the modal features between different data segments.

[0154] In the pre-training stage, the method further includes: calculating a modal contrastive learning loss according to the fused text features and the fused visual features of each sample segment of the video sample, the modal contrastive learning loss being used to increase the similarity of the fused text features and the fused visual features of the same sample segment, and reduce the similarity between the fused text features and the fused visual features of different sample segments.

[0155] For example, in the pre-training stage, the modal contrastive learning loss can be calculated according to the fused text features and the fused visual features of each sample segment of the video sample by using the following formula (2):

[0156] wherein l cma represents the modal contrastive learning loss, n represents the number of data segments into which the video sample is divided, and respectively represent the fused visual feature and the fused text feature of the i th data segment, represents the similarity of the fused visual feature and the fused text feature of the i th data segment. represents the fused text feature of the j th data segment, represents the similarity between the fused visual feature of the i th data segment and the fused text feature of the j th data segment. ε is a pre-configured hyperparameter, and in order to prevent the value of the denominator from being 0, the specific value can be configured according to the needs of the actual application scene, for example, ε can take a value of 0.1.

[0157] In addition, the calculation method of the modal contrastive learning loss can be transformed on the basis of formula (2), for example, replacing with replacing with and the like, or other cross-entropy loss calculation methods can be used to realize, which are not specifically limited here.

[0158] wherein the similarity of the fused visual feature and the fused text feature of the data segment can be realized by calculating the Euclidean distance, cosine similarity, or correlation coefficient between the fused visual feature and the fused text feature, which is not specifically limited here.

[0159] Further, according to the modal contrast learning loss, the parameters of the video theme segmentation model to be trained are adjusted by back propagation, mainly adjusting the parameters of the multi-modal fusion module.

[0160] For example, in the pre-training stage, the cross-entropy loss l vts and the modal contrast learning loss l cma are weighted and summed to obtain a comprehensive loss, and according to the comprehensive loss, the parameters of the video theme segmentation model to be trained are adjusted by back propagation. The weight coefficients of the cross-entropy loss l vts and the modal contrast learning loss l cma may be configured and adjusted according to the needs of the actual application scene, which is not specifically limited here.

[0161] In this embodiment, by adding the modal contrast learning loss in the pre-training stage, a cross-modal alignment task customized for the downstream video theme segmentation task is introduced, and the pseudo-labeled theme boundary is learned using unlabeled data, and the alignment of features of different modalities is enhanced.

[0162] In an optional embodiment, the multi-modal fusion module of the video theme segmentation model includes an attention layer and a hybrid expert layer. The pre-training stage further includes: calculating the balance loss (i.e., l balance ) of the hybrid expert layer; and adjusting the parameters of the hybrid expert layer according to the balance loss. By adding the balance loss function, the problem of uneven load among experts during model training can be solved, which helps each expert module to be fully trained, avoiding some expert modules being overused while other expert modules are relatively idle, thereby avoiding the negative impact of such imbalance on training efficiency and the final performance of the model.

[0163] The balance loss function of the hybrid expert layer includes two parts: an importance loss function (i.e., limportance) and a load loss function (i.e., l load ). Among them, the importance loss function is used to define the importance of each expert module, which is usually reflected by measuring the number of samples processed by each expert module. The purpose of the importance loss function is to encourage all expert modules to participate in training and avoid some expert modules being ignored because they process fewer samples. By adding the importance loss function, the hybrid expert layer will tend to more evenly distribute samples to each expert module, thereby improving the overall training effect. The common calculation method of the load loss function is to calculate the difference between the number of samples processed by each expert and the average number of samples, and to punish it.

[0164] The load loss function is used to solve the problem of uneven sample weight received between the expert modules. The purpose of the load loss function is to ensure that each expert module can receive relatively balanced sample weights, avoiding some expert modules being underestimated because they receive a large number of small-weight samples, and some expert modules being overused because they receive a small number of large-weight samples. A common way to calculate the load loss function is to calculate the variance of the sample weight received by the expert module.

[0165] The balance loss function of the mixed expert layer is the sum of the importance loss function (i.e., limportance) and the load loss function (i.e., l load ). balance load In this embodiment, the balance loss function limportance of the mixed expert layer can be implemented using any existing balance loss function of the mixed expert model, which is not specifically limited here.

[0166] For example, in the pre-training stage, the cross-entropy loss l vts and the balance loss function l balance can be calculated, the cross-entropy loss l vts and the balance loss function l balance are weighted and summed to obtain a comprehensive loss, and according to the comprehensive loss, the parameters of the video topic segmentation model to be trained are adjusted by backpropagation. The weight coefficients of the cross-entropy loss l vts and the balance loss function l balance can be configured and adjusted according to the needs of the actual application scenario, which is not specifically limited here.

[0167] For example, in the pre-training stage, the cross-entropy loss l vts , the modal contrast learning loss l cma and the balance loss function l balance can be calculated, the cross-entropy loss l vts , the modal contrast learning loss l cma and the balance loss function l balance are weighted and summed to obtain a comprehensive loss in the pre-training stage, i.e., l pretrain = l vts + αl cma + βl balance . Wherein, α and β are the weight coefficients of the modal contrast learning loss l cma and the balance loss function l balance , and α and β are both values greater than 0 and less than 1, which can be configured and adjusted according to the needs of the actual application scenario, which is not specifically limited here. Further, according to the comprehensive loss l pretrain ​​The parameters of the video topic segmentation model to be trained are adjusted through back propagation.

[0168] After the pre-training phase is completed, a pre-trained video topic segmentation model is obtained, and the parameters of the video topic segmentation model can be better initialized through the pre-training phase, and in particular, more optimal initialization parameters of the multi-modal fusion module can be obtained.

[0169] In the fine-tuning phase, the pre-trained video topic segmentation model is trained using video samples with annotated topic boundaries to obtain a trained video topic segmentation model.

[0170] In the fine-tuning phase, the pre-trained video topic segmentation model is trained using video samples with annotated topic boundaries as follows:

[0171] The video sample with annotated topic boundaries is divided into multiple sample segments, and the sample segments include text segments and video segments; the multiple sample segments of the video sample are input into the pre-trained video topic segmentation model, the text features and visual features of each sample segment are extracted through the pre-trained video topic segmentation model, the text features and visual features of each sample segment are fused to obtain multi-modal fusion features of each sample segment, and whether each sample segment is a topic boundary is predicted according to the multi-modal fusion features of each sample segment to obtain a topic boundary prediction result of the video sample.

[0172] According to the topic boundary prediction result of the video sample and the annotated topic boundaries of the video sample, a cross-entropy loss is calculated. According to the cross-entropy loss, the parameters of the video topic segmentation model to be trained are adjusted through back propagation. The calculation method of the cross-entropy loss in the fine-tuning phase is similar to that in the pre-training phase, which will not be described here.

[0173] In an optional embodiment, in the fine-tuning phase, when the text features and visual features of each sample segment are fused through the multi-modal fusion module to obtain the multi-modal fusion features of each sample segment, the multi-modal fusion module first fuses the text features and visual features of multiple data segments to obtain fused text features and fused visual features of each data segment; and then the fused text features and fused visual features of each data segment are spliced to obtain the multi-modal fusion features of each data segment.

[0174] In the fine-tuning stage, a cross-modal alignment task similar to the pre-training stage is introduced, and a contrastive learning loss is used to adjust the features learned by the multi-modal fusion module in the model, to maximize the similarity of the fused text features and the fused visual features of the same data segment, while reducing the similarity of the modal features between different data segments. The cross-modal alignment task in the fine-tuning stage is the same as that in the pre-training, and this continuity can ensure consistency of the modalities during pre-training and fine-tuning, thereby promoting consistent interaction between different modalities.

[0175] In the fine-tuning stage, the method further includes: calculating a modal contrastive learning loss according to the fused text features and the fused visual features of each sample segment of the video sample, the modal contrastive learning loss being used to increase the similarity of the fused text features and the fused visual features of the same sample segment, and reduce the similarity between the fused text features and the fused visual features of different sample segments. The calculation method of the modal contrastive loss in the fine-tuning stage is similar to that in the pre-training, which will not be repeated here.

[0176] Further, according to the modal contrastive learning loss, the parameters of the video theme segmentation model to be trained are adjusted through back propagation, mainly the parameters of the multi-modal fusion module.

[0177] For example, in the fine-tuning stage, the cross-entropy loss and the modal contrastive learning loss are weighted and summed to obtain a comprehensive loss, and the parameters of the video theme segmentation model to be trained are adjusted through back propagation according to the comprehensive loss. The weight coefficients of the cross-entropy loss and the modal contrastive learning loss can be configured and adjusted according to the needs of the actual application scenario, which will not be specifically limited here.

[0178] In this embodiment, by adding a modal contrastive learning loss similar to the pre-training stage in the fine-tuning stage, a cross-modal alignment task customized for the downstream video theme segmentation task is introduced, so that the model learns to label the theme boundary and enhance the alignment of different modal features.

[0179] In an optional embodiment, the multi-modal fusion module of the video theme segmentation model includes an attention layer and a hybrid expert layer. The fine-tuning stage further includes: calculating a balance loss (i.e., l balance ) of the hybrid expert layer; and adjusting the parameters of the hybrid expert layer according to the balance loss. By adding the balance loss function, the problem of uneven load between experts during model training can be solved, which helps each expert module to be fully trained, avoids some expert modules from being overused while other expert modules are relatively idle, thereby avoiding the negative impact of such imbalance on training efficiency and the final performance of the model. The calculation method of the balance loss of the hybrid expert layer in the fine-tuning stage is similar to that in the pre-training, which will not be repeated here.

[0180] Exemplarily, in the fine-tuning stage, a cross-entropy loss, a modal contrastive learning loss and a balance loss function can be calculated, and the cross-entropy loss, the modal contrastive learning loss and the balance loss function are weighted and summed to obtain a comprehensive loss in the fine-tuning stage. The weight coefficients of the cross-entropy loss, the modal contrastive learning loss and the balance loss function can be configured and adjusted according to the needs of the actual application scene, which is not specifically limited here. Further, according to the comprehensive loss in the pre-training stage, the parameters of the video topic segmentation model to be trained are adjusted through back propagation.

[0181] In an optional embodiment, a contrastive semantic similarity learning (CSSL) task is introduced in the fine-tuning stage to realize multi-modal feature alignment between topics, so that the model pays more attention to the correlation of enhanced multi-modal representation, and the model can identify the relative consistency relationship within and between topics, which can help the model better capture the relationship and interaction between different topics, thereby improving the performance and accuracy of the model.

[0182] The fine-tuning stage further includes: calculating a semantic similarity contrastive learning loss according to the multi-modal fusion features of each sample segment, the semantic similarity contrastive learning loss being used to increase the similarity of the multi-modal fusion features of different sample segments belonging to the same topic and to reduce the similarity of the multi-modal fusion features of sample segments belonging to different topics.

[0183] Exemplarily, in the fine-tuning stage, the following formula (3) can be used to calculate the semantic similarity contrastive learning loss according to the multi-modal fusion features of each sample segment:

[0184] wherein, l mcssl represents the semantic similarity contrastive learning loss. n represents the number of data segments into which the video sample is divided. m i represents the multi-modal fusion feature of the i-th data segment. k1 represents the number of positive samples, k2 represents the number of negative samples, and k1 and k2 can be configured and adjusted according to the needs of the actual application scene, which is not specifically limited here. represents the multi-modal fusion feature of the j-th positive sample of the i-th data segment. represents the multi-modal fusion feature of the j-th negative sample of the i-th data segment.

[0185] The positive sample of the i-th data segment refers to a data segment having the same theme as the i-th data segment, for example, other data segments in the same video segment as the i-th data segment. The negative sample of the i-th data segment refers to a data segment not having the same theme as the i-th data segment, for example, data segments contained in adjacent video segments of the video segment in which the i-th data segment is located. k1 positive samples are randomly selected from the positive samples of the i-th data segment, and k2 negative samples are randomly selected from the negative samples of the i-th data segment, which are used to calculate the semantic similarity contrast learning loss.

[0186] In addition, the calculation method of the semantic similarity contrast learning loss can be transformed on the basis of formula (3), for example, replacing with replacing with and the like, or other cross-entropy loss calculation methods can be used, which are not specifically limited here.

[0187] The similarity of the multi-modal fusion features of different data segments can be realized by calculating the Euclidean distance, cosine similarity, or correlation coefficient between the multi-modal fusion features of different data segments, which is not specifically limited here.

[0188] Further, according to the semantic similarity contrast learning loss, the parameters of the video theme segmentation model to be trained are adjusted through back propagation.

[0189] Exemplarily, in the pre-training stage, the cross-entropy loss l′ vts , the modality contrast learning loss l′ cma , the semantic similarity contrast learning loss l mcssl , and the balance loss function l′ balance are calculated, and the cross-entropy loss l′ vts , the modality contrast learning loss l′ cma , the semantic similarity contrast learning loss l mcssl , and the balance loss function l′ balance are weighted and summed to obtain the comprehensive loss in the fine-tuning stage, i.e., l finetune = l′ vts + σl′ balance + θl mcssl + γl′ cma . Wherein, σ, θ and γ are weight coefficients of the balance loss function l′ balance , the semantic similarity contrast learning loss l mcssl , and the modality contrast learning loss l′ cma , and σ, θ and γ are values greater than 0 and less than 1, which can be configured and adjusted according to the needs of actual application scenarios, which are not specifically limited here. Further, according to the comprehensive loss lfinetune parameters of the video topic segmentation model to be trained are adjusted through back propagation.

[0190] In the fine-tuning stage, the model pays more attention to the correlation of enhanced multi-modal representations through cross-modal alignment and multi-modal semantic similarity contrast learning tasks, so that the model enhances the consistency between visual features and text features, and can identify the relative consistency relationship within and between topics, thereby improving the quality of video topic segmentation results.

[0191] After the fine-tuning stage is completed, a trained video topic segmentation model is obtained, and the performance of the video topic segmentation model can be improved through the training in the fine-tuning stage.

[0192] Exemplarily, FIG. 9 is a training architecture diagram of a video topic segmentation model provided by an embodiment of the present disclosure. In the pre-training and fine-tuning stages, the loss functions calculated are not exactly the same, but the processing processes of the video samples are the same. As shown in FIG. 9, the video sample is divided into n data segments, including n data segment text segments and video segments

[0193] n data segment text segments are input into a text encoding module, and text features of each text segment are extracted and output by the text encoder. The output features of the text encoding module are projected to a unified dimension by a projection layer, to obtain text features {t1,…,t n} of each data segment.

[0194] n data segment video segments are input into a visual encoding module, and visual features of each video segment are extracted by encoding the input n video segments by the visual encoding module. The output features of the visual encoding module are projected to a unified dimension by a projection layer, to obtain visual features {v1,…,v n} of each data segment.

[0195] Further, the text features {t1,…,t n} and the visual features {v1,…,v n} of the n data segments are input into a multi-modal fusion module (including M multi-modal fusion layers stacked in sequence, as shown as “Mx” in FIG. 9). The text features and the visual features of the n data segments input into the multi-modal fusion module are fused by the multi-modal fusion module, to obtain fusion text features and fusion visual features

[0196] The fused text features and fused visual features of each data segment are concatenated to obtain the multimodal fusion features of each data segment. The multimodal fusion features of n data segments are represented as {m1,…,m n}

[0197] Furthermore, the multimodal fusion features {m1,…,m} of n data segments n The input is given to the classification prediction module, which predicts the probability that each data segment is a topic boundary, denoted as {p1,…,p}. n}, thus obtaining the topic boundary prediction results for the video samples, i.e., {p1,…,p n}

[0198] During the pre-training phase, the topic boundary prediction results {p1,…,p} of the video samples are used. n}, and the topic boundaries of the video samples, to calculate the cross-entropy loss (as shown in Figure 9). vts Based on the fused textual and fused visual features of each video sample segment, the modality contrastive learning loss is calculated (as shown in Figure 9). cma The purpose of this modality contrastive learning loss is to enhance the fusion of textual and visual features of the same sample segment (as shown in Figure 9). cma The similarity between the solid arrows below) and the reduction of the fusion text features and fusion visual features of different sample fragments (as shown in Figure 9) are also considered. cma The similarity between (as indicated by the dashed arrow below).

[0199] During the fine-tuning phase, it is still necessary to calculate the topic boundary prediction results {p1,…,p} based on the video samples. n}, and the topic boundaries of the video samples, to calculate the cross-entropy loss (as shown in Figure 9). vts Based on the fused textual and fused visual features of each video sample segment, the modality contrastive learning loss is calculated (as shown in Figure 9). cma Additionally, during the fine-tuning phase, a semantic similarity contrastive learning loss was added (as shown in Figure 9). mcssl The purpose of semantic similarity contrastive learning loss is to enhance the multimodal fusion features of different sample fragments on the same topic (as shown in Figure 9). mcssl The similarity between the samples (shown by the solid arrows below) is reduced to decrease the multimodal fusion features of sample fragments from different topics (as shown in Figure 9). mcssl The similarity between (as indicated by the dashed arrow below).

[0200] In addition, when the multimodal fusion module of the video topic segmentation model includes an attention layer and a hybrid expert layer, the balance loss of the hybrid expert layer can also be calculated during the pre-training and fine-tuning stages, as not shown in the figure.

[0201] In this embodiment, an initial video topic segmentation model is constructed based on pre-trained visual and text encoders, and the initial video topic segmentation model is continuously pre-trained to obtain a pre-trained video topic segmentation model. In the pre-training (i.e., continuous pre-training) stage, the use of cross-modal alignment tasks and downstream task-related topic segmentation tasks helps to improve the results of the model after fine-tuning. In the fine-tuning stage, the same cross-modal alignment tasks and downstream task-related topic segmentation tasks as in the pre-training stage are used. This continuity can ensure consistency of the modalities during pre-training and fine-tuning, thereby facilitating consistent interaction between different modalities. In addition, a multi-modal semantic similarity contrast learning task is added in the fine-tuning stage. Through the cross-modal alignment tasks and multi-modal semantic similarity contrast learning tasks in the fine-tuning stage, the model enhances the consistency between visual and text features, and can identify the relative consistency relationship within and between topics, accurately capture and utilize complementary information between different modalities, improve the overall consistency of information, and thus improve the quality of video topic segmentation results.

[0202] The video topic segmentation model trained by the method of this embodiment can perform better in scenarios where complex and ambiguous topic boundaries need to be identified, and can more accurately identify topic boundaries, improving the accuracy and quality of video topic segmentation.

[0203] The method of the present disclosure can be applied to online education systems, video conference assistants, and other application scenarios. When implementing corresponding system functions, the video data in the system can be subjected to video topic segmentation. The following example illustrates the application of the method to the video conference assistant scenario, which performs video topic segmentation on conference videos.

[0204] FIG. 10 is a flowchart of a video segmentation method according to an example embodiment of the present disclosure. The execution subject of this embodiment is the server in the system architecture described above. As shown in FIG. 10, the method comprises the following steps:

[0205] In step S1001, a recorded conference video is obtained.

[0206] When applied to the video conference assistant scenario, the recorded conference video usually involves multiple different topics, such as different issues of the conference. By performing topic segmentation on the conference video, the conference video can be segmented into multiple video segments according to different conference issues, and adjacent video segments correspond to different conference issues, which helps relevant personnel quickly locate and access the video segments corresponding to the issues of interest.

[0207] The video conference assistant can send the recorded conference video to the server after the video conference ends. The server receives the conference video sent by the terminal device where the video conference assistant is located.

[0208] For example, the video conference assistant sends the conference video to the server in real time after the conference video recording is completed, triggering the server to perform video topic segmentation on the conference video. Alternatively, the video conference assistant can transmit the conference videos recorded in a time interval in batches to the server at an interval. The server receives a batch of conference videos and performs video topic segmentation on each conference video in the batch respectively.

[0209] Step S1002, the conference video is cut into a plurality of data segments, and the data segments include text segments and video segments.

[0210] Step S1003, the plurality of data segments are input into a video topic segmentation model, text features and visual features of each data segment are extracted through the video topic segmentation model, and whether each data segment is a topic boundary is predicted according to the text features and the visual features of each data segment, to obtain a topic boundary of the conference video.

[0211] Step S1004, according to the topic boundary of the conference video, the video data is segmented into a plurality of video segments, wherein adjacent video segments correspond to different topics.

[0212] In this embodiment, the implementation principle of steps S1002-S1004 for performing video topic segmentation on the conference video is the same as the related content of steps S201-S203 in the foregoing embodiments, and is not limited herein.

[0213] Step S1005, outputting information of the plurality of video segments of the conference video.

[0214] The server can also generate information of the plurality of video segments segmented from the conference video, including but not limited to: generating a topic and / or a content abstract corresponding to each video segment, determining a timestamp (including a start timestamp and / or an end timestamp) of each video segment, and determining a start frame of each video segment.

[0215] Further, the server can output the information of the plurality of video segments segmented from the conference video through a front-end interface. The information of the video segments includes at least one of the following: a timestamp, a start video frame, a topic, and a content abstract.

[0216] The user can select a video segment of interest according to the topic, the content abstract, and other information of each video segment, and quickly find the video segment of interest for playing according to the timestamp and other information of the video segment of interest through fast forward, fast backward, dragging a progress bar, and the like.

[0217] Optionally, the user can select the information of the video segment of interest to himself, such as clicking on the conference topic of interest, to select the video segment corresponding to the conference topic to jump to play. In response to the selection operation on any video segment of the conference video, the conference video is played from the selected video segment.

[0218] The method of the embodiment can better capture the relationship and interaction between different modalities and obtain higher-quality multi-modal fusion features of each data segment by cutting the conference video into multiple data segments, including text segments and video segments, extracting features of a single modality of each data segment, including text features of a text modality and visual features of a visual modality, by a video topic segmentation model, and then fusing features of multiple modalities. Further, according to the multi-modal fusion features of each data segment, whether each data segment is a topic boundary can be accurately predicted, and the topic boundary of the conference video is obtained, improving the accuracy of topic boundary recognition. Further, according to the topic boundary of the conference video, the video data is segmented into multiple video segments, improving the accuracy and quality of the video topic segmentation result.

[0219] FIG. 11 is a structural schematic diagram of a server provided by an embodiment of the present disclosure. As shown in FIG. 11, the server includes a memory 1101 and a processor 1102. The memory 1101 is configured to store computer execution instructions and can be configured to store other various data to support operations on the server. The processor 1102 is in communication connection with the memory 1101 and is configured to execute the computer execution instructions stored in the memory 1101 to implement the technical solutions provided by any of the method embodiments described above, and the specific functions and technical effects that can be achieved are similar, which will not be described here again.

[0220] Optionally, as shown in FIG. 11, the server further includes a firewall 1103, a load balancer 1104, a communication component 1105, a power supply component 1106, and other components. Only some components are shown in FIG. 11, which does not mean that the server only includes the components shown in FIG. 11. In FIG. 11, only a cloud server deployed in the cloud is taken as an example for illustrative purposes, and the server can also be deployed locally, which is not limited here.

[0221] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores computer execution instructions. When the processor executes the computer execution instructions, the method of any of the preceding embodiments is implemented, and the specific functions and technical effects that can be achieved will not be described here again.

[0222] The embodiments of the present disclosure further provide a computer program product comprising a computer program which, when executed by a processor, implements the method of any of the preceding embodiments. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. The at least one processor executes the computer program to enable the server to perform the technical solutions provided by any of the method embodiments, and the specific functions and technical effects that can be achieved are not repeated here.

[0223] The embodiments of the present disclosure provide a chip comprising a processing module and a communication interface, the processing module being capable of executing the technical solutions of the server in the preceding method embodiments. Optionally, the chip further comprises a storage module (e.g., a memory), the storage module being configured to store instructions, and the processing module being configured to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solutions provided by any of the method embodiments.

[0224] The integrated modules in the form of software functional modules described above can be stored in a computer readable storage medium. The software functional modules described above are stored in a storage medium, and include a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method of various embodiments of the present disclosure.

[0225] It should be understood that the processor described above can be a central processing unit (CPU), a graphics processing unit (GPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in at least one processor.

[0226] The storage can include a high-speed Random Access Memory (RAM), and can also include a non-volatile storage, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc. The above-mentioned storage can be an Object Storage Service (OSS). The above-mentioned storage can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read Only Memory (EEPROM), an Erasable Programmable Read Only Memory (EPROM), a Programmable Read Only Memory (PROM), a Read Only Memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk.

[0227] The communication component is configured to facilitate wired or wireless communication between the device in which the communication component is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second generation mobile communication system (2G), a third generation mobile communication system (3G), a fourth generation mobile communication system (4G) / Long Term Evolution (LTE), a fifth generation mobile communication system (5G), or other mobile communication networks, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared technology, Ultra Wide Band (UWB) technology, Bluetooth technology, and other technologies.

[0228] The power component provides power to various components of the device in which the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power component is located. The storage medium can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic storage, flash memory, magnetic or optical disks. The storage medium can be any available media that can be accessed by a general or special purpose computer.

[0229] An example storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can be a part of the processor. Consistent with the teachings provided herein, the processor can execute instructions embodied by the storage medium. The instructions may, and often do, alter the state of the processor, thus causing the processor to perform one or more operations (e.g., one or more operations to implement example embodiments of the disclosure).

[0230] It should be noted that, as used in this document, the terms "includes," "including," "has," "having," "contains," "containing," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by "comprises a... " does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0231] The sequence of the above-mentioned embodiments of the disclosure is only for description, and does not represent the advantages and disadvantages of the embodiments. In addition, in some of the processes described in the above embodiments and drawings, a plurality of operations appear in a specific order, but it should be clearly understood that these operations can be executed in the order they appear in this document or in parallel, and only for distinguishing different operations. The sequence itself does not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent the order. "First" and "second" are different types. The meaning of "multiple" is more than two, unless otherwise specifically limited.

[0232] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be through hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods of various embodiments of the present disclosure.

[0233] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such modifications and equivalents as come within the scope of the present disclosure.

[0234] The above is only the preferred embodiment of the present disclosure, and does not limit the patent scope of the present disclosure, and any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present disclosure, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present disclosure.

Claims

1. A method of video segmentation, wherein, The method comprises the following steps: cutting the video data to be segmented into a plurality of data segments, wherein the data segments comprise text segments and video segments; inputting the plurality of data segments into a video topic segmentation model, extracting text features and visual features of each data segment through the video topic segmentation model, fusing the text features and visual features of each data segment, obtaining multi-modal fusion features of each data segment, and predicting whether each data segment is a topic boundary according to the multi-modal fusion features of each data segment to obtain the topic boundary of the video data; segmenting the video data into a plurality of video segments according to the topic boundary of the video data, wherein adjacent video segments correspond to different topics.

2. The method of claim 1, wherein, The method comprises the following steps: converting the audio of the video data to be segmented into text to obtain text information corresponding to the audio; cutting the text information into a plurality of text segments, and cutting the video of the video data according to the time stamps of the plurality of text segments to obtain video segments corresponding to each text segment; each text segment and the corresponding video segment constitute a data segment.

3. The method of claim 1 or 2, wherein, The video topic segmentation model comprises a text encoding module, a visual encoding module, a multi-modal fusion module, and a classification prediction module. The method comprises the following steps: extracting text features of the text segments of each data segment through the text encoding module; extracting visual features of the video segments of each data segment through the visual encoding module; fusing the text features and visual features of each data segment through the multi-modal fusion module to obtain fused text features and fused visual features of each data segment; splicing the fused text features and fused visual features of each data segment to obtain multi-modal fusion features of each data segment; classifying and predicting whether each data segment is a topic boundary according to the multi-modal fusion features of each data segment through the classification prediction module to obtain the topic boundary of the video data.

4. The method of claim 3, wherein, The multi-modal fusion module comprises an attention layer and a mapping layer, The method comprises the following steps: performing attention calculation on the text features and visual features of the plurality of data segments through the attention layer to obtain multi-modal fusion representations of each data segment; mapping the multi-modal fusion representations of each data segment into fused text features and fused visual features through the mapping layer.

5. The method of claim 3, wherein, The multi-modal fusion module comprises an attention layer and a hybrid expert layer, The multi-modal fusion module fuses the text features and the visual features of the plurality of data segments to obtain fused text features and fused visual features of each data segment, including: The attention layer performs attention calculation on the text features and the visual features of the plurality of data segments to obtain multi-modal fusion representations of each data segment. The hybrid expert layer selects at least one expert module based on a hybrid expert mechanism to map the multi-modal fusion representations of each data segment into fused text features and fused visual features.

6. The method of claim 5, wherein, The hybrid expert layer includes a gating module, a merging module, and a plurality of expert modules. The hybrid expert layer selects at least one expert module based on a hybrid expert mechanism to map the multi-modal fusion representations of each data segment into fused text features and fused visual features, including: The gating module selects at least one expert module based on the input multi-modal fusion representations of each data segment. The at least one expert module maps the multi-modal fusion representations of each data segment into fused text features and fused visual features of each data segment, respectively. The merging module merges the outputs of the at least one expert module based on the weights of the at least one expert module to obtain the fused visual features of each data segment.

7. The method of any one of claims 1-6, wherein, Also including: Outputting information of the plurality of video segments, the information of the video segments including at least one of the following: timestamp, starting video frame, theme, content summary; In response to a selection operation on any video segment, playing the video data starting from the selected video segment.

8. The method of any one of claims 1-6, wherein, The training process of the video theme segmentation model includes: In the pre-training stage, using video samples with unknown theme boundaries, constructing video samples with pseudo-labeled theme boundaries, and using video samples with pseudo-labeled theme boundaries to train the initial video theme segmentation model to obtain a pre-trained video theme segmentation model; In the fine-tuning stage, using video samples with labeled theme boundaries to train the pre-trained video theme segmentation model to obtain a trained video theme segmentation model.

9. The method of claim 8, wherein, In the pre-training stage, using video samples with unknown theme boundaries to construct video samples with pseudo-labeled theme boundaries, including: Cutting the video samples with unknown theme boundaries into a plurality of cut segments; For any video sample with unknown theme boundaries, inserting at least one cut segment not belonging to the video sample into the video sample, and / or replacing at least one cut segment of the video sample with a cut segment not belonging to the video sample to construct a new video sample, and determining the pseudo-labeled theme boundaries of the new video sample.

10. The method of claim 9, wherein, The cutting of the video samples with unknown theme boundaries into a plurality of cut segments includes: Obtaining the duration distribution of different themes; For any video sample with unknown theme boundaries, sampling a duration according to the duration distribution, and cutting the video sample into a plurality of cut segments according to the sampled duration.

11. The method of claim 9, wherein, In the pre-training stage and the fine-tuning stage, a video sample with theme boundaries is used to train a video theme segmentation model to be trained, including: cutting the video sample with theme boundaries into multiple sample segments, the sample segments including text segments and video segments; inputting the multiple sample segments of the video sample into the video theme segmentation model to be trained, extracting text features and visual features of each sample segment through the video theme segmentation model to be trained, fusing the text features and the visual features of each sample segment to obtain multi-modal fusion features of each sample segment, and predicting whether each sample segment is a theme boundary according to the multi-modal fusion features of each sample segment to obtain theme boundary prediction results of the video sample with theme boundaries; calculating a cross-entropy loss according to the theme boundary prediction results of the video sample and theme boundaries of the video sample; adjusting parameters of the video theme segmentation model to be trained according to the cross-entropy loss.

12. The method of claim 11, wherein, The fusing of the text features and the visual features of each sample segment to obtain the multi-modal fusion features of each sample segment includes: fusing the text features and the visual features of the multiple data segments to obtain fused text features and fused visual features of each data segment; splicing the fused text features and the fused visual features of each data segment to obtain the multi-modal fusion features of each data segment.

13. The method of claim 12, wherein, The pre-training stage and the fine-tuning stage further include: calculating a modal contrast learning loss according to the fused text features and the fused visual features of each sample segment of the video sample, the modal contrast learning loss being used to increase similarity of the fused text features and the fused visual features of the same sample segment and to reduce similarity between the fused text features and the fused visual features of different sample segments; adjusting the parameters of the video theme segmentation model to be trained according to the modal contrast learning loss.

14. The method of claim 11, wherein, The fine-tuning stage further includes: calculating a semantic similarity contrast learning loss according to the multi-modal fusion features of each sample segment, the semantic similarity contrast learning loss being used to increase similarity of the multi-modal fusion features of different sample segments belonging to the same theme and to reduce similarity of the multi-modal fusion features of sample segments belonging to different themes; adjusting the parameters of the video theme segmentation model to be trained according to the semantic similarity contrast learning loss.

15. The method of claim 11, wherein, The multi-modal fusion module of the video theme segmentation model includes an attention layer and a hybrid expert layer, and the pre-training stage and the fine-tuning stage further include: calculating a balance loss of the hybrid expert layer; adjusting parameters of the hybrid expert layer according to the balance loss.

16. A video partitioning method, wherein, including: acquiring a recorded conference video; cutting the conference video into multiple data segments, the data segments including text segments and video segments; inputting the multiple data segments into a video theme segmentation model, extracting text features and visual features of each data segment through the video theme segmentation model, and predicting whether each data segment is a theme boundary according to the text features and the visual features of each data segment to obtain theme boundaries of the conference video; According to a subject boundary of the conference video, video data is segmented into a plurality of video segments, wherein adjacent video segments correspond to different subjects; Output information of the plurality of video segments of the conference video.

17. A server, wherein, Comprise: At least one processor; And A memory connected in communication with the at least one processor; Wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the server to perform the method of any one of claims 1-16.

18. A computer readable storage medium, wherein, The computer readable storage medium stores computer executable instructions, and when the processor executes the computer executable instructions, the method of any one of claims 1-16 is implemented.

19. A computer program product comprising a computer program, wherein, The computer program is executed by the processor to implement the method of any one of claims 1-16.

Images