Video clip method, device and storage medium based on multi-modal semantic fusion

Abstract

This invention relates to the field of video editing, and discloses a video editing method, device, and storage medium based on multimodal semantic fusion. The method includes: aligning a sequence of text statements and a sequence of image event descriptions to generate an aligned dual-modal sequence; analyzing the modal overlap in the aligned dual-modal sequence; performing semantic fusion processing on the text elements and image event description elements corresponding to the modal overlap to generate a fused dual-modal sequence; receiving editing text requirements and performing semantic recognition processing on the editing text requirements to generate editing requirements; performing editing planning processing on the fused dual-modal sequence to generate an editing strategy; and editing video data according to the editing strategy to generate an edited video. In this embodiment, the automatic matching of semantically relevant segments in the fused dual-modal sequence significantly improves editing efficiency and intelligence, and substantially enhances the accuracy and completeness of semantic understanding of video content.

Description

Technical Field

[0001] This invention relates to the field of video editing, and more particularly to a video editing method, device, and storage medium based on multimodal semantic fusion. Background Technology

[0002] With the rapid development of short videos and video content creation, the demand for extracting key segments from long videos and generating short videos is increasing. Traditional video editing methods mainly rely on manual annotation and editing, which has the following problems: 1. Manual processing is inefficient and struggles to handle massive amounts of video content; 2. Reliance on the experience of professional editors leads to poor editing consistency; 3. Understanding of video content is often limited to a single modality (e.g., only audio or only visuals), making it difficult to fully grasp the semantic information of the video; 4. Lack of intelligent editing point recognition and scheme generation mechanism.

[0003] While deep learning-based video analysis methods have seen some development in the field, most solutions suffer from the following shortcomings: (1) Single-modal analysis methods, which analyze based only on visual features or only on audio features, cannot fully utilize the multimodal information in the video; (2) Insufficient semantic understanding depth. Traditional methods rely on shallow feature extraction, which makes it difficult to understand the deep semantics and plot logic of videos. The editing scheme generation lacks intelligence and cannot automatically generate editing schemes with coherence and narrative logic according to user needs.

[0004] Therefore, given the current technical problem that video editing solutions rely on a single modality and cannot understand the semantic plot of videos, a new technology is needed to solve this problem. Summary of the Invention

[0005] The main objective of this invention is to solve the technical problem that current video editing solutions have a single analysis modality and cannot understand the semantic plot of videos.

[0006] The first aspect of this invention provides a video editing method based on multimodal semantic fusion, the video editing method comprising: Receive video data to be edited; The video data is subjected to audio extraction processing to obtain an audio stream, and the audio stream is subjected to recognition and segmentation processing to obtain a sequence of sentence texts; Visual event analysis is performed on the video data to obtain an image event description sequence; Based on the timestamps of the video data, the statement text sequence and the image event description sequence are aligned to generate an aligned dual-mode sequence, and the modal overlap in the aligned dual-mode sequence is analyzed. The aligned dual-mode sequence includes: statement text elements corresponding to the statement text sequence and image event description elements corresponding to the image event description sequence. When the modal overlap is greater than a preset overlap threshold, semantic fusion processing is performed on the sentence text element and image event description element corresponding to the modal overlap to generate a fused dual-modal sequence. The system receives the user's editing text request and performs semantic recognition processing on the editing text request based on a preset large language model to generate editing requirements. Based on the preset dynamic programming algorithm and the editing requirements, the fused dual-mode sequence is processed for editing planning to generate an editing strategy; According to the editing strategy, the video data is edited to generate an edited video.

[0007] Optionally, in a first implementation of the first aspect of the present invention, the step of analyzing the modal overlap in the aligned dual-mode sequence includes:

[0008] , among which, T j start Let T be the start timestamp of the i-th statement text element in the statement text sequence. j end Let τ be the end timestamp of the i-th statement text element in the statement text sequence. j start Let τ be the start timestamp of the j-th image event description element in the image event description sequence. j end Let be the end timestamp of the j-th image event description element in the image event description sequence, and overlap(i,j) be the modal overlap between the i-th statement text element and the j-th image event description element, where i and j are positive integer subscripts.

[0009] Optionally, in a second implementation of the first aspect of the present invention, the step of semantically fusing the sentence text elements and image event description elements corresponding to the modal overlap to generate a fused dual-modal sequence includes the following steps:

[0010] , where C k For semantic fusion elements, t k start t is the start timestamp of the modal overlap duration. k endS is the end timestamp of the modal overlap duration. i For the i-th statement text element, D j Let j be the image event description element, overlap(i,j) be the modal overlap between the i-th statement text element and the j-th image event description element, θ be the overlap threshold, Φ() be the fusion function, and k be a positive integer subscript.

[0011] Optionally, in a third implementation of the first aspect of the present invention, the editing requirements include: theme keywords, style preferences, and target duration; the step of performing editing planning processing on the fused dual-mode sequence based on a preset dynamic programming algorithm and the editing requirements to generate an editing strategy includes: Based on the preset matching algorithm and the topic keywords, style preferences, and target duration, the element similarity of the fused dual-mode sequence is calculated to obtain the similarity corresponding to each element in the fused dual-mode sequence. Based on the preset similarity target function and the similarity corresponding to each element, the fused dual-mode sequence is processed to perform optimal segment combination processing to generate an editing strategy.

[0012] Optionally, in a fourth implementation of the first aspect of the present invention, the step of calculating the element similarity of the fused dual-mode sequence based on a preset matching algorithm, the topic keywords, the style preference, and the target duration, to obtain the similarity corresponding to each element in the fused dual-mode sequence, includes:

[0013] , of which M y To fuse the y-th element in the dual-mode sequence, score(M y ) represents the similarity of the y-th element in the fused bimodal sequence, sim() is the similarity function, Nt is the topic keyword, Ns is the style preference, and M y，t To fuse the y-th statement text element in the dual-mode sequence, M y，t To fuse the y-th image event description element in the dual-mode sequence, position(y) is the position weight function of the y-th element in the fused dual-mode sequence, where α, β, and γ are weight coefficients, and α+β+γ=1.

[0014] Optionally, in a fifth implementation of the first aspect of the present invention, the step of performing optimal segment combination processing on the fused dual-mode sequence according to a preset similarity target function and the similarity corresponding to each element to generate an editing strategy includes:

[0015] subject to: ∑time(M)y )*x y ≤t targer ; x y ∈{0,1}; If x y =1 and x y+1 =1, then t y end =t y+1 start Where Y is the total number of elements in the fused dual-mode sequence, and score(M) y To fuse the similarity of the y-th element in the dual-mode sequence, time(M) y ) represents the segment duration of the y-th element in the fused dual-mode sequence, t target For the target duration, x y For the decision of whether to choose to fuse the y-th element in the dual-mode sequence, t y end To merge the end timestamp of the y-th element in the dual-mode sequence, t y+1 start This is the start timestamp of the (y+1)th element in the merged dual-mode sequence.

[0016] Optionally, in a sixth implementation of the first aspect of the present invention, the step of performing recognition and segmentation processing on the audio stream to obtain a sequence of sentence texts includes: Based on a preset ASR model, the audio stream is processed for speech recognition to generate a timestamped text sequence; Based on punctuation marks, semantic segmentation is performed on the timestamped text sequence to generate a sentence text sequence.

[0017] Optionally, in a seventh implementation of the first aspect of the present invention, the step of performing visual event analysis processing on the video data to obtain an image event description sequence includes: The video data is subjected to frame extraction processing according to a preset sampling frequency to obtain video frame data; Based on a preset VLLM model, visual language analysis is performed on the video frame data to generate an image event description sequence.

[0018] A second aspect of the present invention provides a video editing device for multimodal semantic fusion, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the video editing device for multimodal semantic fusion to perform the aforementioned video editing method for multimodal semantic fusion.

[0019] A third aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described multimodal semantic fusion video editing method.

[0020] In this embodiment of the invention, by simultaneously extracting audio and analyzing visual events from video data, a sequence of sentence text and a sequence of image event descriptions are obtained. Alignment processing based on timestamps generates an aligned dual-modal sequence, achieving spatiotemporal correlation between the speech and visual modalities. Furthermore, through modal overlap analysis, semantic fusion processing is performed on highly overlapping regions to generate a fused dual-modal sequence, effectively solving the problem of existing technologies relying solely on a single modality. A preset large language model is used to perform semantic recognition processing on the user's editing text requirements, automatically generating structured editing requirements. This eliminates the need for users to possess professional editing knowledge or perform cumbersome parameter settings. It can understand the user's natural language intent and automatically match semantically relevant segments in the fused dual-modal sequence, significantly improving editing efficiency and intelligence. It fully utilizes the complementary information of the two modalities, addressing the problem of partial semantic understanding and significantly improving the accuracy and completeness of semantic understanding of video content. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of an embodiment of the video editing method using multimodal semantic fusion in this invention. Figure 2 This is a schematic diagram of a specific embodiment of the 102 steps of the multimodal semantic fusion video editing method in this invention; Figure 3 This is a schematic diagram of a specific embodiment of the 103 steps of the multimodal semantic fusion video editing method in this invention; Figure 4 This is a schematic diagram of a specific embodiment of the 107 steps of the multimodal semantic fusion video editing method in this invention. Figure 5 This is a schematic diagram of one embodiment of a video editing device for multimodal semantic fusion in this invention. Detailed Implementation

[0022] This invention provides a video editing method, device, and storage medium based on multimodal semantic fusion.

[0023] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0024] In the description of the embodiments disclosed in this invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0025] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 A schematic diagram of an embodiment of the video editing method for multimodal semantic fusion in this invention includes the following steps: 101. Receive video data to be edited; In this embodiment, the system can first receive 20 minutes of video data to be edited uploaded by the user from the client.

[0026] 102. Perform audio extraction processing on the video data to obtain an audio stream, and perform recognition and segmentation processing on the audio stream to obtain a sentence text sequence; In this embodiment, the audio stream is first extracted from the video data using a single-channel FFmpeg sampling rate of 44100. Then, the speech segments of the audio stream are identified, resulting in a text sequence with start and end timestamps for each speech segment. Finally, this text sequence is semantically segmented according to punctuation and speech segments to generate a sentence text sequence. The sequence elements within the sentence text sequence are complete sentences, each with its start and end timestamps.

[0027] For details, please refer to Figure 2 , Figure 2 This is a schematic diagram of a specific embodiment of step 102 of the video editing method for multimodal semantic fusion in this invention. Step 102 includes the following specific implementation methods: 1021. Based on a preset ASR model, perform speech recognition processing on the audio stream to generate a text sequence with timestamps; 1022. Based on punctuation marks, perform semantic segmentation on the timestamped text sequence to generate a sentence text sequence.

[0028] In steps 1021-1022, the audio stream is first converted into text using an ASR model to generate a text sequence with a start and end timestamp for each text.

[0029] Based on punctuation marks and speech segments, the timestamped text sequence is segmented and decomposed to generate a sentence text sequence. Each sentence text in the sentence text sequence has a start timestamp and an end timestamp.

[0030] 103. Perform visual event analysis processing on the video data to obtain an image event description sequence; In this embodiment, the video data is analyzed frame by frame, and frames are extracted at a sampling rate of 44100. The extracted frame sequence is then subjected to visual understanding to generate an image event description sequence. Each image event description element has a start timestamp and an end timestamp, and each image event description element is a natural language description of each image.

[0031] For details, please refer to Figure 3 , Figure 3 This is a schematic diagram of a specific embodiment of step 103 of the video editing method for multimodal semantic fusion in this invention. Step 103 includes the following specific implementation methods: 1031. According to the preset sampling frequency, the video data is subjected to frame extraction processing to obtain video frame data; 1032. Based on the preset VLLM model, perform visual language analysis on the video frame data to generate an image event description sequence.

[0032] In steps 1031-1032, the sampling frequency is 44100. Based on the sampling frequency, the video data is subjected to frame extraction processing to obtain video frame data.

[0033] By using the VLLM model, also known as the Visual Language Model, visual language analysis is performed on video frame data to obtain the descriptive text for each image event, as well as the start and end timestamps of each image event. The descriptive text for each image event is then organized to generate a sequence of image event descriptions.

[0034] 104. Based on the timestamp of the video data, align the statement text sequence and the image event description sequence to generate an aligned dual-mode sequence, and analyze the modal overlap in the aligned dual-mode sequence, wherein the aligned dual-mode sequence includes: statement text elements corresponding to the statement text sequence and image event description elements corresponding to the image event description sequence; In this embodiment, based on the timestamps of the video data, the statement text sequence and the image event description sequence are mapped and associated. The statement text sequence and the image event description sequence are aligned on the timeline to generate an aligned dual-mode sequence. This aligned dual-mode sequence contains statement text sequences and image event description sequences with aligned timestamps. Furthermore, the statement text sequence has statement text elements ordered by time, and the image event description sequence has image event description elements ordered by time.

[0035] Then, the temporal overlap between each element in the statement text sequence and the image event description sequence in the aligned dual-mode sequence is analyzed.

[0036] Specifically, step 104, "analyzing the modal overlap in the aligned dual-mode sequence," includes the following specific implementation methods:

[0037] , among which, T j start Let T be the start timestamp of the i-th statement text element in the statement text sequence. j end Let τ be the end timestamp of the i-th statement text element in the statement text sequence. j start Let τ be the start timestamp of the j-th image event description element in the image event description sequence. j end Let be the end timestamp of the j-th image event description element in the image event description sequence, and overlap(i,j) be the modal overlap between the i-th statement text element and the j-th image event description element, where i and j are positive integer subscripts.

[0038] It should be noted that the value range of overlap(i,j) is set in the interval of 0-1, and it is used to analyze the speech modality and image modality in the homogeneous dual-mode sequence as an alignment index.

[0039] 105. When the modal overlap is greater than a preset overlap threshold, semantic fusion processing is performed on the sentence text element and image event description element corresponding to the modal overlap to generate a fused dual-modal sequence. In this embodiment, when the modal overlap is greater than a preset overlap threshold, the corresponding segments in the aligned dual-mode sequence are semantically fused. If the modal overlap is not greater than the preset overlap threshold, the single-mode related data information in the aligned dual-mode sequence is retained.

[0040] After traversing and analyzing the aligned dual-mode sequences, semantic fusion processing is performed on the elements corresponding to those whose modal overlap is greater than a preset overlap threshold, generating a fused dual-mode sequence arranged in ascending order of timestamps.

[0041] Specifically, step 105 includes the following specific implementation methods: The step of semantically fusing the sentence text elements and image event description elements corresponding to the modal overlap to generate a fused dual-modal sequence includes the following steps:

[0042] , where C k For semantic fusion elements, t k start t is the start timestamp of the modal overlap duration. k end S is the end timestamp of the modal overlap duration. i For the i-th statement text element, D j Let j be the image event description element, overlap(i,j) be the modal overlap between the i-th statement text element and the j-th image event description element, θ be the overlap threshold, Φ() be the fusion function, and k be a positive integer subscript.

[0043] It should be noted that C k It is the fused data of a single fused segment, while the unfused segments retain their original modal data.

[0044] 106. Receive the user's editing text request, and perform semantic recognition processing on the editing text request according to the preset large language model to generate editing requirements; In this embodiment, the editing request is received from the user's natural language input, and the intent recognition and key information extraction are performed using a large language model to generate the editing request.

[0045] 107. Based on the preset dynamic programming algorithm and the editing requirements, perform editing planning processing on the fused dual-mode sequence to generate an editing strategy; In this embodiment, based on the extracted editing requirements, semantic similarity is calculated for each segment in the fused dual-mode sequence. After calculating the similarity between each segment in the fused dual-mode sequence and the editing requirements, the optimal combination of segments is selected for editing requirements through similarity analysis using a dynamic programming algorithm.

[0046] Specifically, the editing requirements include: theme keywords, style preferences, and target duration. Please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of a specific embodiment of the 107 steps of the multimodal semantic fusion video editing method in this invention. The 107 steps include the following specific implementation methods: 1071. Based on the preset matching algorithm and the topic keywords, style preferences, and target duration, perform element similarity calculation on the fused dual-mode sequence to obtain the similarity corresponding to each element in the fused dual-mode sequence; 1072. Based on the preset similarity objective function and the similarity corresponding to each element, the fused dual-mode sequence is subjected to optimal segment combination processing to generate an editing strategy.

[0047] In steps 1071-1072, the similarity is calculated by using topic keywords, style preferences, and target duration as the basis for comparison. The element similarity of the fused dual-mode sequence is calculated to obtain the similarity of each element in the fused dual-mode sequence.

[0048] Based on the similarity, an objective function is designed, and the similarity of each element is substituted into the objective function. Based on the design constraints, the fused dual-mode sequence is processed to achieve optimal segment combination. Based on the optimization results, a structured editing strategy is generated. The editing strategy can be JSON text as follows: EditPlan = { "videos": [ { "video_name": "output_1", "segments": [ { "source_video": "input.mp4", "start_time": "HH:MM:SS,fff", "end_time": "HH:MM:SS,fff", "description": "Semantic description of the fragment"}, ... ] }, ... ] } Specifically, step 1071 includes the following specific implementation methods:

[0049] , of which M y To fuse the y-th element in the dual-mode sequence, score(M y ) represents the similarity of the y-th element in the fused bimodal sequence, sim() is the similarity function, Nt is the topic keyword, Ns is the style preference, and M y，t To fuse the y-th statement text element in the dual-mode sequence, M y，tTo fuse the y-th image event description element in the dual-mode sequence, position(y) is the position weight function of the y-th element in the fused dual-mode sequence, where α, β, and γ are weight coefficients, and α+β+γ=1.

[0050] Specifically, step 1072 includes the following specific implementation methods:

[0051] subject to: ∑time(M) y )*x y ≤t targer ; x y ∈{0,1}; If x y =1 and x y+1 =1, then t y end =t y+1 start Where Y is the total number of elements in the fused dual-mode sequence, and score(M) y To fuse the similarity of the y-th element in the dual-mode sequence, time(M) y ) represents the segment duration of the y-th element in the fused dual-mode sequence, t target For the target duration, x y For the decision of whether to choose to fuse the y-th element in the dual-mode sequence, t y end To merge the end timestamp of the y-th element in the dual-mode sequence, t y+1 start This is the start timestamp of the (y+1)th element in the merged dual-mode sequence.

[0052] It should be noted that the objective function of the design is: The design aims to maximize The data, with the constraint condition: ∑time(M) y )*x y ≤t targer x y ∈{0,1}, if x y =1 and x y+1 =1, then t y end =t y+1 start The goal is to find a clipping method that maximizes the objective function under constraints.

[0053] 108. Based on the editing strategy, the video data is edited to generate an edited video.

[0054] In this embodiment, each segment in the video data is precisely trimmed using FFmpeg according to the editing strategy. All trimmed segments are then concatenated in chronological order of their timestamps to generate the final edited video. It should be noted that using the concat demuxer method for concatenation can ensure seamless splicing and encoding consistency.

[0055] Metadata can be generated to record the source of the fragments and their timeline mapping relationship. The metadata generation method is as follows: ProjectJSON = { "segments": [ { "timeline": { "showTime": [show_start_i, show_end_i], / / Time in the final video "resourceTime": [res_start_i, res_end_i] / / Time in the original video }, "source": {"filename": "...", "filepath": "..."} }, ... ] } In one specific implementation, a 60-minute interview video (interview.mp4) is input, and the user's request is to "generate three short videos of about one minute each, highlighting the guest's core viewpoints." The ASR model then processes and identifies the complete question-and-answer dialogue, totaling 126 sentences and approximately 8000 words. The VLLM model identifies 35 complete discussion topics, including "trends in artificial intelligence development," "discussions on technological ethics," and "future prospects."

[0056] The ASR sentences are time-aligned with VLLM events, and fragments with high matching scores are marked as "high-quality fragments". A comprehensive analysis SRT file containing 126 comprehensive records is generated.

[0057] Analyzing user needs: Extracting keywords "core viewpoint," quantity "3," and duration "1 minute." Calculating semantic similarity to filter 15 candidate segments, and using dynamic programming to optimize and select the 3 optimal segment combinations. Video 1: 00:08:15-00:09:22 (Discussion on AI security) Video 2: 00:23:40-00:24:51 (Viewpoints on technological innovation) Video 3: 00:47:10-00:48:05 (Outlook for future development) The system automatically trims three segments to 67 seconds, 71 seconds, and 55 seconds, generating three independent MP4 files and a project JSON file that records the complete timeline mapping.

[0058] The entire process took about 8 minutes (including 3 minutes for ASR, 4 minutes for VLLM, and 1 minute for editing and generation). The three generated short videos were semantically complete, had a smooth rhythm, and met the user's needs.

[0059] In this embodiment of the invention, by simultaneously extracting audio and analyzing visual events from video data, a sequence of sentence text and a sequence of image event descriptions are obtained. Alignment processing based on timestamps generates an aligned dual-modal sequence, achieving spatiotemporal correlation between the speech and visual modalities. Furthermore, through modal overlap analysis, semantic fusion processing is performed on highly overlapping regions to generate a fused dual-modal sequence, effectively solving the problem of existing technologies relying solely on a single modality. A preset large language model is used to perform semantic recognition processing on the user's editing text requirements, automatically generating structured editing requirements. This eliminates the need for users to possess professional editing knowledge or perform cumbersome parameter settings. It can understand the user's natural language intent and automatically match semantically relevant segments in the fused dual-modal sequence, significantly improving editing efficiency and intelligence. It fully utilizes the complementary information of the two modalities, addressing the problem of partial semantic understanding and significantly improving the accuracy and completeness of semantic understanding of video content.

[0060] Figure 5 This is a schematic diagram of the structure of a multimodal semantic fusion video editing device 500 provided in an embodiment of the present invention. The multimodal semantic fusion video editing device 500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 510 and memory 520, and one or more storage media 530 for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the multimodal semantic fusion video editing device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the multimodal semantic fusion video editing device 500.

[0061] The video editing device 500 based on multimodal semantic fusion may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, Free BSD, etc. Those skilled in the art will understand that... Figure 5 The illustrated structure of the video editing device based on multimodal semantic fusion does not constitute a limitation on the video editing device based on multimodal semantic fusion. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0062] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the multimodal semantic fusion video editing method.

[0063] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0064] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0065] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video editing method based on multimodal semantic fusion, characterized in that, Including the following steps: Receive video data to be edited; The video data is subjected to audio extraction processing to obtain an audio stream, and the audio stream is subjected to recognition and segmentation processing to obtain a sentence text sequence; Visual event analysis is performed on the video data to obtain an image event description sequence; Based on the timestamps of the video data, the statement text sequence and the image event description sequence are aligned to generate an aligned dual-mode sequence, and the modal overlap in the aligned dual-mode sequence is analyzed. The aligned dual-mode sequence includes: statement text elements corresponding to the statement text sequence and image event description elements corresponding to the image event description sequence. When the modal overlap is greater than a preset overlap threshold, semantic fusion processing is performed on the sentence text element and image event description element corresponding to the modal overlap to generate a fused dual-modal sequence. The system receives the user's editing text request and performs semantic recognition processing on the editing text request based on a preset large language model to generate editing requirements. Based on the preset dynamic programming algorithm and the editing requirements, the fused dual-mode sequence is processed for editing planning to generate an editing strategy; The video data is edited according to the editing strategy to generate an edited video.

2. The video editing method based on multimodal semantic fusion according to claim 1, characterized in that, The step of analyzing the modal overlap in the aligned dual-mode sequence includes: Among them, T j start Let T be the start timestamp of the i-th statement text element in the statement text sequence. j end Let τ be the end timestamp of the i-th statement text element in the statement text sequence. j start Let τ be the start timestamp of the j-th image event description element in the image event description sequence. j end Let be the end timestamp of the j-th image event description element in the image event description sequence, and overlap(i,j) be the modal overlap between the i-th statement text element and the j-th image event description element, where i and j are positive integer subscripts.

3. The video editing method based on multimodal semantic fusion according to claim 2, characterized in that, The step of semantically fusing the sentence text elements and image event description elements corresponding to the modal overlap to generate a fused dual-modal sequence includes the following steps: Among them, C k For semantic fusion elements, t k start t is the start timestamp of the modal overlap duration. k end S is the end timestamp of the modal overlap duration. i For the i-th statement text element, D j Let j be the image event description element, overlap(i,j) be the modal overlap between the i-th statement text element and the j-th image event description element, θ be the overlap threshold, Φ() be the fusion function, and k be a positive integer subscript.

4. The video editing method based on multimodal semantic fusion according to claim 1, characterized in that, The editing requirements include: theme keywords, style preferences, and target duration. The step of performing editing planning processing on the fused dual-mode sequence based on a preset dynamic programming algorithm and the editing requirements to generate an editing strategy includes: Based on the preset matching algorithm and the topic keywords, style preferences, and target duration, the element similarity of the fused dual-mode sequence is calculated to obtain the similarity corresponding to each element in the fused dual-mode sequence. Based on the preset similarity target function and the similarity corresponding to each element, the fused dual-mode sequence is processed to perform optimal segment combination processing to generate an editing strategy.

5. The video editing method based on multimodal semantic fusion according to claim 4, characterized in that, The step of calculating the element similarity of the fused dual-mode sequence based on the preset matching algorithm, the topic keywords, the style preference, and the target duration, to obtain the similarity corresponding to each element in the fused dual-mode sequence includes: Among them, M y To fuse the y-th element in the dual-mode sequence, score(M y ) represents the similarity of the y-th element in the fused bimodal sequence, sim() is the similarity function, Nt is the topic keyword, Ns is the style preference, and M y，t To fuse the y-th statement text element in the dual-mode sequence, M y，t To fuse the y-th image event description element in the dual-mode sequence, position(y) is the position weight function of the y-th element in the fused dual-mode sequence, where α, β, and γ are weight coefficients, and α+β+γ=1.

6. The video editing method based on multimodal semantic fusion according to claim 4, characterized in that, The step of performing optimal segment combination processing on the fused dual-mode sequence based on a preset similarity objective function and the similarity corresponding to each element to generate an editing strategy includes: subject to: ∑time(M y )*x y ≤t targer ; x y ∈{0,1}； If x y =1 and x y+1 =1, then t y end =t y+1 start Where Y is the total number of elements in the fused dual-mode sequence, and score(M) y To fuse the similarity of the y-th element in the dual-mode sequence, time(M) y ) represents the segment duration of the y-th element in the fused dual-mode sequence, t target For the target duration, x y For the decision of whether to choose to fuse the y-th element in the dual-mode sequence, t y end To merge the end timestamp of the y-th element in the dual-mode sequence, t y+1 start This is the start timestamp of the (y+1)th element in the merged dual-mode sequence.

7. The video editing method based on multimodal semantic fusion according to claim 1, characterized in that, The step of performing recognition and segmentation processing on the audio stream to obtain a sequence of sentence text includes: Based on a preset ASR model, the audio stream is processed for speech recognition to generate a timestamped text sequence; Based on punctuation marks, semantic segmentation is performed on the timestamped text sequence to generate a sentence text sequence.

8. The video editing method based on multimodal semantic fusion according to claim 1, characterized in that, The step of performing visual event analysis processing on the video data to obtain an image event description sequence includes: The video data is subjected to frame extraction processing according to a preset sampling frequency to obtain video frame data; Based on a preset VLLM model, visual language analysis is performed on the video frame data to generate an image event description sequence.

9. A video editing device with multimodal semantic fusion, characterized in that, The multimodal semantic fusion video editing device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the multimodal semantic fusion video editing device to perform the multimodal semantic fusion video editing method as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the video editing method of multimodal semantic fusion as described in any one of claims 1-8.

Images