Video segmentation method and device based on multi-modal features, equipment and storage medium

Abstract

The embodiment of the invention provides a multi-modal feature-based video segmentation method. The method comprises the following steps of: respectively extracting image features, audio features and text features of a video frame, voice and a text of the voice; aligning the image features, the audio features and the text features through timestamps; fusing the aligned image features, audio features and text features to obtain a feature sequence with a timestamp; analyzing the feature sequence with the timestamp, and extracting feature mutation points in the feature sequence with the timestamp; filtering the feature mutation points to obtain segmentation points; and segmenting the video data based on the segmentation points to obtain a plurality of video clips. Through multi-modal feature extraction and fusion, video semantic structures are deeply observed, high similarity of fragment contents is ensured, videos are comprehensively and deeply understood, the analysis automation and intelligence level is improved, and it is ensured that results are accurate and timely. The problem of video segmentation is solved, precise self-adaptive segmentation is realized, and the subsequent analysis efficiency and precision are improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of multimodal recognition and related technical fields, and in particular, to a method, apparatus, device and storage medium for video segmentation based on multimodal features. Background Art

[0002] With the development of multimodal recognition technology, the understanding and analysis of video content has gradually become the core cornerstone of content management and distribution systems. Compared with traditional graphic content, videos have higher requirements for analysis technology due to their variable length and dynamic content. Therefore, it is inevitable to use segmentation strategies to conduct in-depth analysis of videos. Given the diversity and uncertainty of video shooting techniques, video segmentation strategies also vary greatly, and these strategies have a profound impact on the final content analysis results.

[0003] However, in the existing technology, video analysis technologies based on deep neural networks have emerged, showing great potential and advantages. However, one of the current technical bottlenecks is that some deep neural network models are limited by network scale and resource consumption, and are difficult to directly apply to the global analysis of the entire video. The video still needs to be segmented in advance.

[0004] Traditional video segmentation methods mostly rely on fixed duration as the basis for division. Although this method is simple and easy to implement, it can easily cause semantically coherent content to be artificially divided, while ignoring the intrinsic connection between semantically similar segments. This not only limits the accuracy of the analysis results, but also hinders the full exploration of the value of video content.

[0005] Therefore, there is an urgent need for a video segmentation method with multimodal features to support video segmentation with multimodal features. Summary of the invention

[0006] The embodiments described herein provide a video segmentation method, apparatus, device, and storage medium based on multimodal features to solve the problems existing in the prior art.

[0007] According to a first aspect of the present disclosure, a video segmentation method based on multimodal features is provided, comprising:

[0008] Extract image features, audio features and text features of video frames, speech and speech text respectively;

[0009] Aligning the image features, the audio features, and the text features by timestamp;

[0010] The aligned image features, audio features and text features are fused to obtain a feature sequence with a timestamp;

[0011] Analyzing the feature sequence with the timestamp, and extracting the feature mutation points in the feature sequence with the timestamp;

[0012] Filter the characteristic mutation points to obtain segmentation points;

[0013] The video data is segmented based on the segmentation points to obtain a plurality of video segments.

[0014] In some embodiments of the present disclosure, before the steps of respectively extracting image features, audio features, and text features of video frames, speech, and speech text, the following steps are included:

[0015] determining the video frame and the speech from video data;

[0016] A text of the speech is determined.

[0017] In some embodiments of the present disclosure, the step of determining video frames and speech from video data includes:

[0018] Extracting the video frame from the video data;

[0019] The speech in the video data is separated.

[0020] In some embodiments of the present disclosure, the step of determining the text of the speech includes:

[0021] converting the speech into segmented text;

[0022] Processing the segmented text to determine the text representing the speech specifically includes: presetting an upper limit value for the number of paragraph characters, and a cumulative paragraph list for storing the currently accumulated paragraph content, traversing the segmented text to be processed, and performing the following processing on each segmented text: checking whether the length of the currently accumulated paragraph content has reached or exceeded the preset upper limit value for the number of paragraph characters, if so, saving the currently accumulated paragraph as a new final segment, and clearing the cumulative paragraph list to prepare for the next round of accumulation, if the length of the currently accumulated paragraph content has not reached the preset upper limit value for the number of paragraph characters, directly adding the currently segmented text to the cumulative paragraph list, starting from the beginning of the cumulative paragraph list, removing paragraph items one by one, checking the length of the remaining accumulated paragraph content after each removal, and stopping the removal operation when the length of the accumulated paragraph content is less than half of the preset upper limit value for the number of paragraph characters for the first time, and combining all the final segmented texts obtained in the above process together as the text representing the speech.

[0023] In some embodiments of the present disclosure, the aligning of the multimodal features by timestamps includes:

[0024] Select a preset interval, find the features of a certain mode within the interval, and perform pooling operations as the features of the mode within the interval.

[0025] In some embodiments of the present disclosure, the step of fusing the aligned image features, audio features, and text features specifically utilizes a neural network to perform feature fusion, converting the input features of multiple modalities into one feature;

[0026] Wherein, the neural network is obtained by training based on a pre-selected feature extraction network;

[0027] The neural network includes two parts. The first part aligns the features of multiple modalities into the same space to obtain concatenated high-dimensional aligned features. The second part maps the high-dimensional aligned features into a lower-dimensional space.

[0028] The training of the neural network is divided into two parts. First, the first part is trained using a multimodal contrastive learning model, and then the weight of the first part is frozen, and a contrastive learning model is used to keep the output result of the second part in a spatial distribution similar to that of the output result of the first part.

[0029] In some embodiments of the present disclosure, analyzing the feature sequence with the timestamp to extract the feature mutation point in the feature sequence with the timestamp includes:

[0030] Using a kernel function to calculate the similarity between every two features, constructing a similarity matrix, wherein the elements of the similarity matrix represent the similarity between the features;

[0031] Calculating a cumulative similarity matrix, wherein the cumulative similarity matrix represents the cumulative similarity from the beginning of the video to each feature;

[0032] Use dynamic programming to calculate the optimal segmentation cost;

[0033] Based on the optimal segmentation cost, reverse tracing is performed to find the characteristic mutation point.

[0034] In some embodiments of the present disclosure, filtering the characteristic mutation points to obtain the segmentation points includes:

[0035] Using preset rules to select required or appropriate feature mutation points from the feature mutation points as segmentation points, and removing unnecessary or inappropriate feature mutation points;

[0036] The preset rules include: setting a minimum length threshold for each video segment, setting a threshold according to feature similarity to determine the feature similarity between adjacent segments, and filtering the feature mutation points that do not meet the preset rules.

[0037] According to a second aspect of the present disclosure, a video segmentation device based on multimodal features is provided, comprising:

[0038] A first extraction module is used to extract image features, audio features and text features of video frames, speech and speech text respectively;

[0039] an alignment module, configured to align the image features, the audio features, and the text features through timestamps;

[0040] A fusion module is used to fuse the aligned image features, audio features, and text features to obtain a feature sequence with a timestamp;

[0041] A second extraction module is used to analyze the feature sequence with the timestamp and extract the feature mutation point in the feature sequence with the timestamp;

[0042] A filtering module, used for filtering the characteristic mutation points to obtain segmentation points;

[0043] The segmentation module is used to segment the video data based on the segmentation points to obtain multiple video segments.

[0044] In some embodiments of the present disclosure, it further includes:

[0045] A first determination module, configured to determine the video frame and the voice from video data;

[0046] The second determination module is used to determine the text of the speech.

[0047] According to a third aspect of the present disclosure, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method in any one of the above embodiments are implemented.

[0048] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0049] The video segmentation method, device, equipment and storage medium based on multimodal features provided by the embodiments of the present disclosure respectively extract the image features, audio features and text features of video frames, voices and voice texts; align the image features, audio features and text features by timestamps; fuse the image features, audio features and text features after alignment to obtain a feature sequence with a timestamp; analyze the feature sequence with a timestamp to extract the feature mutation points in the feature sequence with a timestamp; filter the feature mutation points to obtain segmentation points; segment the video data based on the segmentation points to obtain multiple video clips. The present application can intelligently identify the multimodal features in the video, and accordingly realize accurate and adaptive segmentation, which greatly promotes the deep mining and effective utilization of video content. The present application extracts and fuses multimodal features to deeply understand the semantic structure of the video, ensure that the content of the clips is highly similar, fully and deeply understand the video, improve the level of analysis automation and intelligence, and ensure that the results are accurate and timely. Support user-defined segmentation to adapt to various scenarios. Introduce a recursive algorithm to detect feature mutations, process videos of any length, and finely divide the content. Solve the problem of video segmentation, realize accurate and adaptive segmentation, and improve the efficiency and accuracy of subsequent analysis.

[0050] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below. It should be noted that the drawings described below only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure, wherein:

[0052] Figure 1 It is a flowchart of a video segmentation method based on multimodal features provided by an embodiment of the present disclosure;

[0053] Figure 2 It is a schematic diagram of the overall process of a video segmentation method based on multimodal features provided by an embodiment of the present disclosure;

[0054] Figure 3 is a structural schematic diagram of a video segmentation device based on multimodal features provided by an embodiment of the present disclosure;

[0055] Figure 4 It is a structural diagram of a computer device provided in an embodiment of the present disclosure.

[0056] In the drawings, reference numerals with the same last two digits correspond to the same elements. It should be noted that the elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work also fall within the scope of protection of the present disclosure.

[0058] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase "embodiments" in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0059] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists, A and B exist at the same time, and B exists. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0060] Furthermore, in all embodiments of the present disclosure, terms such as “first” and “second” are used only to distinguish one component (or a part of a component) from another component (or another part of a component).

[0061] In the description of the present application, unless otherwise specified, "plurality" means more than two (including two), and similarly, "multiple groups" means more than two groups (including two).

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0063] Based on the problems existing in the existing technology, Figure 1 is a flow chart of a video segmentation method based on multimodal features provided by an embodiment of the present disclosure, such as Figure 1 As shown in FIG. 1 , the specific process of the video segmentation method based on multimodal features includes:

[0064] Step S110, extracting image features, audio features and text features of the video frame, the speech and the text of the speech respectively;

[0065] As a specific feasible implementation method, feature extraction is performed on the video frame, the speech and the voice using a suitable neural network to obtain image features, audio features and text features;

[0066] The video frame, the speech, and the feature extraction network used by the speech need to match the subsequent feature fusion network, that is, the extracted image features, audio features, and text features meet the requirements of the fusion network;

[0067] As a specific possible implementation method, for each data type, an appropriate feature extraction method is used, including:

[0068] Perform feature extraction on image data to obtain image features. For example, CNN can be used to extract visual features from images.

[0069] Perform feature extraction on audio data to obtain audio features. For example, MFCC can be used to extract audio features.

[0070] Perform feature extraction on text data to obtain text features. For example, you can use the bag-of-words model or Word2Vec to extract text features.

[0071] In a specific implementation, as an example, before step S110, the following is further included:

[0072] Step S1001, determining video frame data and voice data from video data;

[0073] Acquire video data, and extract video frames, audio, and text from the video data;

[0074] The video data is multimodal video data, which is video data containing images, audio and text at the same time.

[0075] In a specific implementation, as an example, step S1001 includes:

[0076] Step S10011, obtain video data.

[0077] Step S10012, extracting the video frame from the video data;

[0078] The video frame is a still image in a video sequence, which is the basic unit of a video. A video consists of continuous video frames, each of which represents a complete image at a certain point in time. Video is created by continuously capturing or generating a series of consecutive image frames. During video playback, these frames are played continuously at a certain rate, thus producing a dynamic video effect.

[0079] In a specific implementation, as an example, the image content of one frame can be extracted every several frames to form a video frame (image) sequence. For example, for a video data, the image content of one frame can be extracted every 5 frames to form a video frame (image) sequence.

[0080] Step S10013: Separate the voice from the video data.

[0081] In a specific implementation, as an example, the ffmpeg (Fast Forward Moving Picture Experts Group) tool can be used to separate the audio stream in the multimodal video, and the audio stream is stored as an audio file to obtain the voice data in the video.

[0082] Step S1002: determining the text of the speech;

[0083] As a specific implementable method, step S1002 includes:

[0084] Step S10021, converting the speech into segmented text;

[0085] As a specific feasible implementation method, a natural regression neural network is used to convert the audio sequence in the speech into a text sequence. Since the audio-translated text is segmented by pauses and the number of characters by default, this segmentation method will result in more paragraphs and less content in each paragraph, so the segmented data needs to be further processed;

[0086] Step S10022: Process the segmented text to determine the text representing the speech.

[0087] As a specific feasible implementation method, an upper limit M of the number of characters in a paragraph is preset, and an accumulated paragraph list (empty list) is used to store the currently accumulated paragraph content, and the segmented content to be processed is traversed, and the following processing is performed on each segmentation result:

[0088] Check whether the current accumulated paragraph length has reached or exceeded the preset M;

[0089] If the current accumulated paragraph length has reached or exceeded the preset M, the current accumulated paragraph is saved as a new final segment, and the accumulated paragraph list is cleared to prepare for the next round of accumulation;

[0090] If the current accumulated paragraph length has not reached M, then directly add the content of the current segment to the accumulated paragraph list, remove the paragraph items one by one from the beginning of the accumulated paragraph list, and check the length of the remaining accumulated paragraphs after each removal;

[0091] When the length of the accumulated segment is less than half of M (i.e., M / 2) for the first time, the removal operation is stopped. During this process, multiple shorter final segments may be formed;

[0092] All final segments obtained in the above process (including the segments that were initially directly segmented due to reaching the length of the M value, and the segments formed during the adjustment process) are combined together as the final result of the overall processing, that is, the text representing the speech.

[0093] S120: Align the image features, the audio features, and the text features through timestamps.

[0094] Specifically, in the timestamp dimension, three features are aligned: the image feature, the audio feature, and the text feature, to ensure that they correspond to data at the same time point in the time dimension.

[0095] For image features, you can choose to use image processing technology (such as convolutional neural network) to extract the visual features of key frames while considering the continuity information between video frames;

[0096] For audio features, MFCC and other techniques can be used to extract audio features and ensure that they are aligned with the video frame image features of the corresponding timestamp;

[0097] For text features, natural language processing technology (such as bag-of-words model or Word2Vec) can be used to extract text features, and it is also necessary to ensure that they are aligned with the video frame image features and audio features of the corresponding timestamp.

[0098] In a specific implementation, as an example, since the timestamps corresponding to different features are often inconsistent, a preset interval is selected, and the features of a certain modality within the interval are found, and a pooling operation is performed, which is used as the features of the modality within the interval. For example, a 1-second interval is preset, and the audio features of audio data within 1 second are found, and a pooling operation is performed, which is used as the audio features of the audio data within 1 second. For another example, a 2-second interval is preset, and the image features of image data within 2 seconds are found, and a pooling operation is performed, which is used as the image features of the image data within 2 seconds.

[0099] Among them, the pooling is spatial pooling, which is a method for extracting features in convolutional neural networks. It aggregates and statistically processes different features to obtain relatively lower dimensions while avoiding overfitting. Pooling retains most of the important information while reducing the dimensions of each feature map. Currently, there are mainly maximization, averaging, and summing. The most common pooling operations are mean pooling and max pooling. Mean pooling: Calculate the average value of the image area and use it as the pooled value of the area. Max pooling: Select the maximum value of the image area and use it as the pooled value of the area. Define a spatial neighborhood and take the largest element from the corrected feature map, or take the average value.

[0100] S130, fusing the aligned image features, audio features, and text features to obtain a feature sequence with a timestamp.

[0101] In a specific implementation, as another embodiment, a neural network is used for feature fusion to convert the input features of multiple modalities into one feature. The neural network here is obtained by training based on a pre-selected feature extraction network.

[0102] The network consists of two parts. The first part aligns the features of multiple modalities into the same space to obtain concatenated high-dimensional aligned features. The second part maps the concatenated high-dimensional aligned features into a lower-dimensional space.

[0103] The training of the neural network is divided into two parts (the first part and the second part). First, the multimodal contrastive learning mode is used to train the feature alignment of the first part, and then the weight of the first part is frozen, and the contrastive learning mode is used to make the output result of the second part maintain a spatial distribution similar to the output result of the first part.

[0104] S140: Analyze the feature sequence with the timestamp, and extract feature mutation points in the feature sequence with the timestamp.

[0105] In a specific implementation, for example, the KTS algorithm can be used to analyze the multimodal feature sequence and extract the characteristic mutation points in the multimodal feature sequence. The KTS algorithm (Kullback-Leibler Total Segmentation) is mainly used to detect characteristic mutation points in a time series.

[0106] Specifically, suppose there is a set of feature vectors of video frames ({f1,f2,…,f T}), where T represents the total number of frames of the video;

[0107] Use a kernel function (such as a Gaussian kernel function) to calculate the similarity between every two features and construct a similarity matrix K. The elements of the similarity matrix represent the similarity between the features, and formula (1) is as follows:

[0108]

[0109] Where i and j are both positive integers greater than 0, i represents the i-th feature, j represents the j-th feature, and σ represents the standard deviation of all features;

[0110] Calculate the cumulative similarity matrix S, where S represents the cumulative similarity from the beginning of the video to each feature. Formula (2) is as follows:

[0111]

[0112] Where i and j are both positive integers greater than 0, i represents the i-th feature, j represents the j-th feature, and k and l are subscripts in the summation symbol;

[0113] Dynamic programming is used to calculate the optimal segmentation cost P. Let P(m) represent the optimal segmentation cost of the first m frames of the video, and formula (3) is as follows:

[0114] P(m)=min 1≤k<m [P(k)+S(k+1,m)] (3)

[0115] Wherein, P(0)=0, m is a positive integer greater than 1, and k represents each segment from 1 to m;

[0116] Based on the optimal segmentation cost P, reverse tracking is performed to find the feature mutation point and segment the video, including:

[0117] Once the characteristic mutation point is detected, the reverse tracking method can be used to trace the starting and ending positions of the characteristic mutation point in time. This helps to determine when a change begins and when it ends. According to the detected characteristic mutation points and the results of reverse tracking, the video is segmented and divided into multiple clips or sub-videos, each sub-video corresponding to a video segment from one characteristic mutation point to the next characteristic mutation point.

[0118] S150: Filter the characteristic mutation points to obtain segmentation points.

[0119] In a specific implementation, a preset rule is used to screen out necessary or appropriate characteristic mutation points from the characteristic mutation points as segmentation points, and unnecessary or inappropriate characteristic mutation points are removed.

[0120] Specifically, the preset rules include:

[0121] 1. Set a minimum length threshold for each video segment to ensure that the length of the segmented video segments meets expectations and avoid generating too short video segments. This can be achieved by checking and adjusting the length of the video segments near the segmentation point.

[0122] 2. Set a threshold based on the similarity of features to determine the degree of feature similarity between adjacent paragraphs. If the similarity exceeds the threshold, you can consider merging these adjacent paragraphs to improve the coherence of video segmentation.

[0123] 3. For segmentation points that do not conform to the preset rules, they can be regarded as noise or abnormal situations, and then filtered or adjusted to ensure the accuracy and consistency of the final video segmentation.

[0124] S160: Segment the video data based on the segmentation points to obtain a plurality of video segments.

[0125] Based on the obtained segmentation points, the video data is segmented to obtain a plurality of video segments cut according to the video segmentation points.

[0126] The video segmentation refers to segmenting a video sequence into multiple regions according to certain standards, in order to obtain multiple video clips. Specifically, the video is segmented according to the requirements into multiple video clips.

[0127] The video segmentation method based on multimodal features provided by the embodiment of the present disclosure determines video frames and speech from video data; determines the text of speech; extracts image features, audio features, and text features of video frames, speech, and speech text respectively; aligns image features, audio features, and text features through timestamps; fuses the image features, audio features, and text features after alignment to obtain a feature sequence with a timestamp; analyzes the feature sequence with a timestamp to extract feature mutation points in the feature sequence with a timestamp; filters the feature mutation points to obtain segmentation points; and segments the video data based on the segmentation points to obtain multiple video clips. This application uses multimodal feature extraction and fusion to gain a deep insight into the semantic structure of the video, ensure that the content of the clips is highly similar, fully and deeply understand the video, improve the level of analysis automation and intelligence, and ensure that the results are accurate and timely. Supports user-defined segmentation to adapt to various scenarios. Introduces a recursive algorithm to detect feature mutations, process videos of any length, and finely divide the content. It solves the problem of video segmentation, achieves accurate adaptive segmentation, and improves the efficiency and accuracy of subsequent analysis.

[0128] The overall process of the video segmentation method based on multimodal features of this application can be referred to Figure 2 shown.

[0129] Based on the above embodiments, the present disclosure also provides a video segmentation device based on multimodal features, such as Figure 3 As shown, the video segmentation device based on multimodal features includes:

[0130] A first extraction module 310, for extracting image features, audio features and text features of video frames, speech and speech text respectively;

[0131] An alignment module 320, configured to align the image features, the audio features, and the text features by using timestamps;

[0132] A fusion module 330 is used to fuse the image features, audio features and text features that have been aligned to obtain a feature sequence with a timestamp;

[0133] A second extraction module 340 is used to analyze the feature sequence with a timestamp and extract feature mutation points in the feature sequence with a timestamp;

[0134] A filtering module 350 is used to filter the characteristic mutation points to obtain segmentation points;

[0135] The segmentation module 360 ​​is used to segment the video data based on the segmentation points to obtain multiple video segments.

[0136] In a specific embodiment, it also includes:

[0137] A first determination module 370, configured to determine the video frame and the speech from the video data;

[0138] A second determination module 380, configured to determine the text of the speech;

[0139] In a specific implementation, the first determining module 310 is specifically configured to:

[0140] Extracting the video frame from the video data;

[0141] The speech in the video data is separated.

[0142] In a specific implementation manner, the second determining module is specifically used to:

[0143] converting the speech into segmented text;

[0144] marking pause information in the text;

[0145] Based on the annotated text, determining text representing the speech;

[0146] Process the segmented data, including:

[0147] Preset an upper limit M of the number of paragraph characters and an empty list to store the currently accumulated paragraph content, traverse the segmented content to be processed, and perform the following processing on each segmentation result:

[0148] Check whether the current accumulated paragraph length has reached or exceeded the preset M value;

[0149] If the current accumulated paragraph length has reached or exceeded the preset M value, the current accumulated paragraph is saved as a new final segment, and the accumulated paragraph list is cleared to prepare for the next round of accumulation;

[0150] If the current accumulated paragraph length has not reached the M value, the content of the current segment is directly added to the accumulated paragraph list, and paragraph items are removed one by one from the beginning of the accumulated paragraph list. After each removal, the length of the remaining accumulated paragraphs is checked;

[0151] When the length of the accumulated segment is less than half of M (i.e., M / 2) for the first time, the removal operation is stopped. During this process, multiple shorter final segments may be formed;

[0152] All final segments obtained in the above process (including the segments that were directly segmented initially due to reaching the length of the M value, and the segments formed during the adjustment process) are combined together as the final result of the overall processing.

[0153] In a specific implementation, the alignment module is specifically used to:

[0154] Select a preset interval, find the features of a certain mode within the interval, and perform pooling operations as the features of the mode within the interval.

[0155] In a specific implementation, the fusion module is specifically used to:

[0156] A neural network is used for feature fusion to convert multiple input features into one feature; the neural network is trained based on a pre-selected feature extraction network;

[0157] The neural network includes two parts. The first part aligns the features of multiple modalities to the same space, and the second part maps the spliced ​​high-dimensional aligned features to a lower-dimensional space. The training of the neural network is divided into two parts. First, the first part is trained using a multimodal contrastive learning model, and then the weight of the first part is frozen, and a contrastive learning model is used to ensure that the output result of the second part maintains a spatial distribution similar to the output result of the first part.

[0158] In a specific implementation, the second extraction module is specifically used to:

[0159] Using a kernel function to calculate the similarity between every two features, constructing a similarity matrix, wherein the elements of the similarity matrix represent the similarity between the features;

[0160] Calculating a cumulative similarity matrix, wherein the cumulative similarity matrix represents the cumulative similarity from the beginning of the video to each feature;

[0161] Use dynamic programming to calculate the optimal segmentation cost;

[0162] Based on the optimal segmentation cost P, reverse tracking is performed to find the feature mutation point and segment the video.

[0163] In a specific implementation, the filtering module is specifically used for:

[0164] After obtaining the segmentation points, the preset rules are used to screen these segmentation points and remove unnecessary or inappropriate segmentation points;

[0165] The preset rules include: setting a minimum length threshold for each video segment; setting a threshold according to the similarity of the features to determine the degree of feature similarity between adjacent segments; and treating segmentation points that do not meet the preset rules as noise or abnormal situations, and then filtering or adjusting them.

[0166] The video segmentation device based on multimodal features provided by the embodiment of the present disclosure can deeply understand the semantic structure of the video through multimodal feature extraction and fusion, ensure that the content of the segments is highly similar, fully and deeply understand the video, improve the level of analysis automation and intelligence, and ensure that the results are accurate and timely. It supports user-defined segmentation and adapts to various scenarios. It introduces a recursive algorithm to detect feature mutations, process videos of any length, and finely divide the content. It solves the problem of video segmentation, realizes accurate and adaptive segmentation, and improves the efficiency and accuracy of subsequent analysis.

[0167] The computer device includes a memory 410 and a processor 420 that are connected to each other through a system bus. It should be noted that the figure only shows a computer device with components 410-420, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0168] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0169] The memory 410 includes at least one type of readable storage medium, and the readable storage medium includes a non-volatile memory or a volatile memory, for example, a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc., and the RAM may include a static RAM or a dynamic RAM. In some embodiments, the memory 410 may be an internal storage unit of a computer device, for example, a hard disk or a memory of the computer device. In other embodiments, the memory 310 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card or a flash card (FlashCard) equipped on the computer device. Of course, the memory 410 may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory 410 is generally used to store the operating system and various application software installed on the computer device, such as the program code of the above method. In addition, the memory 410 may also be used to temporarily store various data that have been output or are to be output.

[0170] The processor 420 is generally used to perform the overall operation of the computer device. In this embodiment, the memory 410 is used to store program codes or instructions, the program code includes computer operation instructions, and the processor 420 is used to execute the program codes or instructions stored in the memory 410 or process data, such as running the program code of the above method.

[0171] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0172] Another embodiment of the present application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads a computer-readable program code stored in the computer-readable medium, so that the processor can execute the functional actions specified in each step or a combination of steps in the above method; and generate a device for implementing the functional actions specified in each block or a combination of blocks in the block diagram.

[0173] Computer-readable media include but are not limited to electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any appropriate combination of the foregoing, the memory is used to store program codes or instructions, the program codes include computer operating instructions, and the processor is used to execute the program codes or instructions of the above methods stored in the memory.

[0174] For the definitions of memory and processor, please refer to the description of the aforementioned computer device embodiment and will not be repeated here.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] Each functional unit or module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0178] Unless the context clearly indicates otherwise, the singular form of the words used herein and in the appended claims includes the plural and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the words "comprise" and "include" are to be interpreted as inclusive rather than exclusive. Likewise, the terms "include" and "or" should be interpreted as inclusive unless such interpretation is expressly prohibited herein. Where the term "example" is used herein, particularly when it is located after a group of terms, the "example" is merely exemplary and illustrative and should not be considered exclusive or comprehensive.

[0179] Further aspects and scopes of adaptability become apparent from the description provided herein. It should be understood that various aspects of the present application can be implemented individually or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are intended for purposes of illustration only and are not intended to limit the scope of the present application.

[0180] Several embodiments of the present disclosure are described in detail above, but it is obvious that those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The protection scope of the present disclosure is defined by the attached claims.

Claims

1. A video segmentation method based on multimodal features, characterized in that: include: Extract image features, audio features and text features of video frames, speech and speech text respectively; Aligning the image features, the audio features, and the text features by timestamp; The aligned image features, audio features and text features are fused to obtain a feature sequence with a timestamp; Analyzing the feature sequence with the timestamp, and extracting the feature mutation points in the feature sequence with the timestamp; Filter the characteristic mutation points to obtain segmentation points; The video data is segmented based on the segmentation points to obtain a plurality of video segments.

2. The method according to claim 1, characterized in that Before the steps of respectively extracting the image features, audio features and text features of the video frame, the speech and the text of the speech, the steps include: determining the video frame and the speech from video data; A text of the speech is determined.

3. The method according to claim 2, characterized in that The step of determining the video frame and the voice from the video data comprises: Extracting the video frame from the video data; The speech in the video data is separated.

4. The method according to claim 2, characterized in that: The step of determining the text of the speech comprises: converting the speech into segmented text; Processing the segmented text to determine the text representing the speech specifically includes: presetting an upper limit value for the number of paragraph characters, and a cumulative paragraph list for storing the currently accumulated paragraph content, traversing the segmented text to be processed, and performing the following processing on each segmented text: checking whether the length of the currently accumulated paragraph content has reached or exceeded the preset upper limit value for the number of paragraph characters, if so, saving the currently accumulated paragraph as a new final segment, and clearing the cumulative paragraph list to prepare for the next round of accumulation, if the length of the currently accumulated paragraph content has not reached the preset upper limit value for the number of paragraph characters, directly adding the currently segmented text to the cumulative paragraph list, starting from the beginning of the cumulative paragraph list, removing paragraph items one by one, checking the length of the remaining accumulated paragraph content after each removal, and stopping the removal operation when the length of the accumulated paragraph content is less than half of the preset upper limit value for the number of paragraph characters for the first time, and combining all the final segmented texts obtained in the above process together as the text representing the speech.

5. The method according to claim 1, characterized in that The aligning process of the multimodal features by using timestamps includes: Select a preset interval, find the features of a certain mode within the interval, and perform pooling operations as the features of the mode within the interval.

6. The method according to claim 1, characterized in that The step of fusing the aligned image features, audio features and text features specifically comprises using a neural network to perform feature fusion and converting the input features of multiple modalities into one feature; Wherein, the neural network is obtained by training based on a pre-selected feature extraction network; The neural network includes two parts. The first part aligns the features of multiple modalities into the same space to obtain concatenated high-dimensional aligned features. The second part maps the high-dimensional aligned features into a lower-dimensional space. The training of the neural network is divided into two parts. First, the first part is trained using a multimodal contrastive learning model, and then the weight of the first part is frozen, and a contrastive learning model is used to keep the output result of the second part in a spatial distribution similar to that of the output result of the first part.

7. The method according to claim 1, characterized in that The filtering of the characteristic mutation points to obtain the segmentation points includes: Using preset rules to select required or appropriate feature mutation points from the feature mutation points as segmentation points, and removing unnecessary or inappropriate feature mutation points; The preset rules include: setting a minimum length threshold for each video segment, setting a threshold according to feature similarity to determine the feature similarity between adjacent segments, and filtering the feature mutation points that do not meet the preset rules.

8. A video segmentation device based on multimodal features, characterized in that: include: A first extraction module is used to extract image features, audio features and text features of video frames, speech and speech text respectively; an alignment module, configured to align the image features, the audio features, and the text features through timestamps; A fusion module is used to fuse the aligned image features, audio features, and text features to obtain a feature sequence with a timestamp; A second extraction module is used to analyze the feature sequence with the timestamp and extract the feature mutation point in the feature sequence with the timestamp; A filtering module, used for filtering the characteristic mutation points to obtain segmentation points; The segmentation module is used to segment the video data based on the segmentation points to obtain multiple video segments.

9. A computer device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images