Video clip method and system based on multi-modal large model cooperation

Abstract

The application discloses a video clip method based on multimodal large model cooperation and belongs to the technical field of video processing. Specifically, the method comprises the following steps: S1, video preprocessing and intelligent blocking: preprocessing the original video, cutting the video into small blocks, ensuring that the end of each small block is a complete sentence, outputting the timestamps and audio text segments of each block and storing them in structured data; S2, multimodal scene separation and verification: inputting the audio text segments and their timestamps into a language large model, initially separating scenes according to rules and reconstructing a sprite, inputting the sprite into a visual large model to verify whether the goods are switched, and finally outputting the scene time, title and synopsis. Through the combination strategy of "audio feature extraction + semantic integrity judgment + quantitative blocking rule", the problems of sentence truncation and timestamp disorder that are prone to occur in traditional video blocking are solved. Specifically, the sound intensity, speech pause features and language large model-recognized sentence boundaries are combined.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video clipping method and system based on multi-modal large model cooperation. BACKGROUND

[0002] With the rapid development of live commerce, a large amount of live video needs to be quickly clipped into highlight segments to meet the secondary dissemination needs. Traditional manual clipping has problems such as low efficiency, high cost, and non-uniform standards. The emergence of automatic clipping effectively alleviates the above problems. Existing automatic clipping methods mostly rely on a single modality (such as only based on audio or vision), and have defects such as inaccurate product segmentation, out-of-sync highlight extraction and picture, and imprecise material matching. SUMMARY

[0003] The purpose of the present application is to provide a video clipping method and system based on multi-modal large model cooperation, which can solve the problem of inaccurate product segmentation, out-of-sync highlight extraction and picture, and imprecise material matching caused by the dependence of existing automatic clipping methods on a single modality.

[0004] TECHNICAL SOLUTION: To solve the above technical problems, according to one aspect of the present application, more specifically, a video clipping method and system based on multi-modal large model cooperation, specifically comprising the following steps:

[0005] S1, video preprocessing and intelligent blocking: preprocessing the original video, cutting the video into small blocks, ensuring that the end of each small block is a complete sentence, outputting the timestamps, audio text segments of each block and storing them in structured data;

[0006] S2, multi-modal scene separation and verification: inputting each block, i.e. audio text segment and its timestamp, into a language large model, initially dividing scenes according to rules and constructing a sprite, inputting a visual large model to verify whether the product has switched, if the product has not switched, reinitializing the division, verifying again, and dynamically adjusting and circulating to finally output scene time, title and synopsis;

[0007] S3, product highlight sentence extraction and sorting: inputting the scene text into a language large model, extracting highlight sentences in combination with multi-dimensional highlight information, introducing audience interaction data and product type preset templates to adjust sentence weights, and sorting according to weights to form a highlight segment timeline;

[0008] S4, product close-up positioning and insertion: dividing, marking and numbering the scene video, inputting it into a visual large model to obtain the grid number where the product is located to position the product, dynamically adjusting the close-up parameters according to the product type, and inserting the close-up into the corresponding interval of the highlight sequence through fade-in and fade-out transitions;

[0009] S5, intelligent keyword extraction and material matching: extract keywords and sort them by frequency and importance, generate text descriptions containing details, uses, and emotions for each material, convert keywords and material descriptions into vectors, match similar materials with high similarity, and prioritize matching corresponding dynamic materials when keywords appear frequently.

[0010] S6, video integration packaging and adaptive output: integrate main track, close-up track, and material track content, adjust style and length according to output platform, detect picture occlusion before output, and adjust, finally output video and metadata.

[0011] Further, the step S1 specifically includes the following steps:

[0012] S11, audio feature extraction: audio separation of the original video, extraction of sound intensity sequence and speech pause features;

[0013] S12, semantic integrity judgment: convert audio to text by calling a speech recognition model, analyze sentence boundaries of the text by a language large model, and detect time points corresponding to punctuation marks including periods and exclamation marks;

[0014] S13, block rule execution: combine sound intensity, low intensity segment as breakpoint and sentence end time, cut video into small blocks with a preset time range, and ensure that the end of each small block is a complete sentence;

[0015] S14, block result storage: output the start and end timestamps of each block and the corresponding audio text segment, and store them as structured data.

[0016] Further, the step S2 specifically includes the following steps:

[0017] S21, input each block text of the block into a language large model, perform scene division based on preset rules, and have the following strategies for the division results: if no goods are divided or only one good is divided, it is considered that the goods explanation has not ended, and the next subtitle data will be spliced; if two goods are divided and the second goods explanation does not exceed two minutes, it is considered that it has not ended, delay for two minutes to ensure stable division; if two goods are divided and the second goods explanation exceeds two minutes, it is considered that the first goods explanation has ended, output the first division result, and continue to splice the second goods subtitle; if multiple division results appear, only the last two results are retained, the rest are output; if multiple short division results less than two minutes appear, only the last two results are retained, the rest are merged into one result and named as "complex scene";

[0018] S22, the video of a certain preset time before and after the segmentation point of the preliminary segmentation result is extracted frame image at the frequency of one frame per second, spliced in time sequence as a sprite with horizontal arrangement, frame spacing 10 pixels;

[0019] S23, input the sprite into the visual large model to identify the product switching feature, if the visual confirms the product switching, keep the initial segmentation result;

[0020] S24, if not switched, extend the segmentation point backward according to the continuity of the buffer picture, re-execute steps S21-S23 until the visual confirms the accuracy of segmentation;

[0021] S25, output the scene text including the start and end time, scene title and content introduction of each scene.

[0022] Further, the step S3 specifically comprises the following steps:

[0023] S31, input the scene text into the language large model, extract the highlight sentence combined with the preset highlight dimension, output the start and end time and content of the sentence;

[0024] S32, introduce auxiliary data to optimize the highlight sorting, including combining the barrage keywords and like peak time during live broadcast, increasing the audience interaction weight of the sentence mentioning the corresponding content, and presetting the commodity type weight of the commodity type according to the sorting template;

[0025] S33, according to the weight calculation result, sort the highlight sentences according to the template order to form the highlight segment basic timeline.

[0026] Further, when performing intelligent keyword extraction and material matching, the language large model and precision prompt word combination are used to extract keywords: input the scene text and highlight sentence, set the prompt word as "extract product name, core buying point, preferential information and emotional tendency", sort the extracted keywords according to the appearance frequency and preset importance, pre-process the material according to "detail description + purpose + emotional label" to generate description text, convert the keywords and material text description into vectors, calculate the matching degree through cosine similarity, select the top three materials with the highest matching degree, if the appearance frequency of the preset keyword is greater than the preset frequency, preferentially match the dynamic sticker of the countdown animation, and insert the matched material into the corresponding position according to the timeline.

[0027] Further, in step S6, when performing video integration packaging and adaptive output, the close-up track and the main track are superimposed, and the material track is inserted according to the timestamp. According to the preset style parameters and time length parameters of the output platform, the picture is detected by the visual large model before output to determine whether there is an occlusion. If there is an occlusion, the material position is automatically adjusted. The final video and metadata are output. The close-up track is a close-up picture of the product. The main track is a video segment picture corresponding to a highlight sentence. The material track is a matched material.

[0028] According to another aspect of the present application, a video clipping system based on multi-modal large model cooperation is provided, characterized in that: the system is used to realize the above-mentioned video clipping method based on multi-modal large model cooperation, and includes: a video preprocessing and intelligent blocking module, a multi-modal scene separation and verification module, a product highlight sentence extraction and sorting module, a product visual close-up positioning and insertion module, an intelligent keyword extraction and material matching module, and a video integration packaging and adaptive output module.

[0029] The video preprocessing and intelligent blocking module is used for preprocessing the original video, cutting the video into small blocks, ensuring that the end of each small block is a complete sentence, outputting the timestamp, audio text segment of each block, and storing them in structured data.

[0030] The multi-modal scene separation and verification module is used for inputting each block into a language large model, initially separating scenes according to rules and reconstructing a wizard graph, inputting a visual large model to verify whether the product is switched, and finally outputting scene time, title and synopsis.

[0031] The product highlight sentence extraction and sorting module is used for inputting scene text into a language large model, extracting highlight sentences in combination with multi-dimensional highlight information, introducing audience interaction data and product type preset templates to adjust sentence weights, and sorting highlight segments according to weights to form a highlight timeline.

[0032] The product visual close-up positioning and insertion module is used for dividing, marking and numbering scene videos, inputting a visual large model to obtain the grid number of the product to position the product, dynamically adjusting close-up parameters according to the product type, and inserting the close-up into the corresponding interval of the highlight sequence through fade-in and fade-out transitions.

[0033] The intelligent keyword extraction and material matching module is used for extracting keywords and sorting them according to frequency and importance, generating text descriptions of each material containing details, uses and emotions, converting keywords and material descriptions into vectors, matching the top-ranked materials through similarity, and preferentially matching corresponding dynamic materials at the corresponding time point when the keywords appear frequently.

[0034] Video integration package and adaptive output module: used for integrating main track, close-up track, material track content, adjusting style and length according to output platform, detecting picture occlusion before output and adjusting, finally outputting video and metadata.

[0035] Advantages:

[0036] 1. By the combination strategy of "audio feature extraction + semantic integrity judgment + quantitative block rule", the problems of sentence truncation and timestamp confusion in traditional video block are solved. Specifically, by combining the sound intensity, speech pause features and language model recognized sentence boundaries, it ensures that each block ends with a complete sentence, and the block duration and structured storage standardization provide a unified and standardized input for subsequent language model and visual model processing, avoiding processing errors caused by fragmented or incomplete input.

[0037] 2. Through the multi-modal collaborative mechanism of "language model initial division + visual sprite verification + dynamic adjustment cycle", the problem of inaccurate product segmentation caused by relying on a single mode of audio or vision is solved. After the language model divides the scene based on product name and explanation length, the visual model verifies whether the product has actually switched through the continuous sprite in the time dimension. If it has not switched, it will adjust through the cycle of "extending the segmentation point → re-dividing → verifying again", until the visual confirms the accurate segmentation. This process effectively avoids the mis-segmentation of "the anchor mentions the next product in advance, but the screen does not switch" or "the screen switches, but the audio is not synchronized", improves the accuracy of product scene segmentation, and ensures that each scene corresponds to a complete explanation process of a single product.

[0038] 3. Through the "multi-dimensional highlight extraction + weight optimization sorting" mechanism, the problem of traditional highlight extraction relying only on text keywords and being out of touch with audience interest is solved. On the one hand, the language model extracts highlight sentences based on preset dimensions such as product core selling points and discount information; on the other hand, it introduces audience interaction data and product type templates to adjust sentence weights, so that the highlight sequence contains not only product key information, but also audience focus and cognitive logic.

[0039] 4. Through the "grid marking positioning + dynamic parameter adaptation" technology, the problem of inaccurate product location description by the visual model and the incoordination between close-up and picture is solved. The video frame is divided into 50x50 pixel grids and marked with numbered red dots, and the visual model only needs to output the serial number to accurately locate the product, avoiding the "illusion phenomenon"; at the same time, the close-up parameters are dynamically adjusted according to the product type, and the fade-in and fade-out transition is realized to achieve a smooth transition between close-up and main picture. This process ensures that the product close-up highlights the details without damaging the integrity of the picture, allowing the audience to clearly capture the key features of the product, with visual expression comparable to manual fine editing but with greatly improved efficiency.

[0040] 5、Through the "language large model keyword extraction + material text description vector matching" mechanism, the limitations of traditional TF-IDF algorithm keyword extraction and the problem of material matching interference are overcome. After the language large model extracts precise keywords such as product name and core selling point combined with fine-tuning prompt words, the material is converted into a vector through "detail description + purpose + emotional label" text processing, and then matched through cosine similarity, reducing the interference data of direct image vector matching, and the material matching hit rate is improved by more than 40%. At the same time, when preset keywords appear frequently, the preset stickers are matched first, so that the packaging material and scene content are highly consistent, and the information transmission efficiency and visual appeal of the video are enhanced.

[0041] 6、Through the "multi-track integration + platform adaptive adjustment + output pre-check" mechanism, the problem of single output format and mismatch with platform characteristics of existing editing methods is solved. Multi-track integration ensures that high light segments, product close-ups, and packaging materials are layered and coordinated to avoid picture confusion; adjust the style parameters according to the output platform to make the video conform to the platform transmission characteristics; before output, the visual large model detects picture occlusion and automatically adjusts to ensure the integrity and professionalism of the final video. This process improves the secondary dissemination conversion rate of the video on different platforms by more than 30%, and realizes rapid output within 10 minutes after the live broadcast ends, greatly improving the editing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a method flowchart. DETAILED DESCRIPTION

[0043] In order to make the technical scheme of the present application clearer, the following will further explain the present application in combination with the drawings and specific embodiments.

[0044] EMBODIMENT

[0045] Video editing method based on multi-modal large model cooperation.

[0046] First step, video preprocessing and intelligent block: pre-process the original video, cut the video into small blocks, ensure that the end of each small block is a complete sentence, output the timestamp, audio text segment of each block and store it as structured data. Specifically, the following steps are included:

[0047] 1、Audio feature extraction: separate the audio from the original video, extract the sound intensity sequence and speech pause features, such as the position of the silent segment length ≥ 0.5 seconds.

[0048] 2、Semantic integrity judgment: call the speech recognition model such as volcano ASRapi to convert the audio to text, analyze the sentence boundaries of the text through the language large model, and detect the time points corresponding to the punctuation marks including period, exclamation point, etc.

[0049] 3. Block rule execution: combined with sound intensity, low intensity section is preferred as breakpoint and sentence end time, cutting video into small blocks with preset time range, ensuring that the end of each small block is a complete sentence, such as "the color number of this lipstick will be introduced here".

[0050] 4. Block result storage: output the start and end time stamp of each block and the corresponding audio text segment, stored as structured data such as JSON format.

[0051] Through the combination of "audio feature extraction + semantic integrity judgment + quantitative block rule", the problems of sentence truncation and timestamp confusion in traditional video block are solved. Specifically, combined with sound intensity, speech pause features and language model recognition of sentence boundary, it ensures that the end of each block is a complete sentence, and the block duration and structured storage standardization provide unified and standardized input for subsequent language model and visual model processing, avoiding processing errors caused by fragmented or incomplete input.

[0052] Second step, multi-modal scene separation and verification: input each block of audio text segment and its timestamp into the language model, and reconstruct the sprite according to the rule, input the visual model to verify whether the commodity is switched, and finally output the scene time, title and introduction:

[0053] 1. Input each block text of the block into the language model, such as LLaMA series, and divide the scene based on the preset rules, such as commodity name, function description and other keywords. For the segmentation results, the following strategies are adopted: if the commodity or only one commodity is segmented, it is considered that the commodity explanation is not finished, and the next subtitle data will be spliced; if two commodities are segmented and the second commodity explanation is less than two minutes, it is considered that it has not ended, delay2 minutes to ensure stable segmentation; if two commodities are segmented and the second commodity explanation is more than two minutes, it is considered that the first commodity explanation is finished, and the first segmentation result is output, the second commodity belongs to the subtitle, which continues to splice the next one, if there are multiple segmentation results, only the last two results are retained, the rest is output, if there are multiple continuous short segmentation (less than 2 minutes) results, only the last two results are retained, the rest is merged into one result and named as "complex scene".

[0054] 2. For the 30-second video before and after the preliminary segmentation result, extract frame images at a frequency of one frame per second, splice them into a horizontal arrangement in chronological order, and convert the time dimension to the spatial dimension.

[0055] 3. Input the sprite into the visual model, such as the improved CLIP model, to identify the commodity switching features, such as the main commodity in the picture changing from "lipstick" to "foundation", if the visual confirms the commodity switching, keep the preliminary result.

[0056] 4. If not switched, extend the segmentation point backward for 30-60 seconds according to the continuity of the buffered picture, re-execute steps 1-3 until the visual confirmation of accurate segmentation.

[0057] 5. Output the scene text of each scene including the start and end time, scene title, and content summary.

[0058] Through the multi-modal collaborative mechanism of "language large model initial segmentation + visual wizard graph verification + dynamic adjustment cycle", the problem of inaccurate product segmentation caused by relying on only audio or visual single mode in existing methods is broken through. After the language large model initially segments the scene based on product name, explanation duration, etc., the visual large model verifies whether the product actually switches through the time-dimensionally continuous wizard graph. If not, it adjusts through the cycle of "extending the segmentation point → re-initializing → re-verifying" until the visual confirmation of accurate segmentation. This process effectively avoids the mis-segmentation of "the anchor mentions the next product in advance but the picture does not switch" or "the picture switches but the audio is not synchronized", improves the accuracy of product scene segmentation, and ensures the complete explanation process of each scene corresponding to a single product.

[0059] Third step, product highlight sentence extraction and sorting: input the scene text into the language large model, extract the highlight sentence combined with multi-dimensional highlight information, introduce audience interaction data and product type preset template to adjust the sentence weight, and sort according to the weight to form the highlight segment timeline. Specifically, the following steps are included:

[0060] 1. Input the scene text into the language large model, extract the highlight sentence combined with the preset highlight dimensions such as product core selling points, discount information, and user concerned question answers, output the start and end time of the sentence and the content such as "buy 2 get 1 for this cream, only today", timestamp 00:05:20-00:05:30.

[0061] 2. Introduce auxiliary data to optimize highlight sorting, including combining the live broadcast's bullet screen keywords such as "price" "effect", peak time of likes, increasing the weight of audience interaction weight for the sentence mentioning the corresponding content, and sorting according to the preset sorting template of product type such as clothing: style → material → matching → discount; electronic product: function → performance → price → after-sales).

[0062] 3. Sort the highlight sentences according to the weight calculation results according to the template order to form the highlight segment basic timeline.

[0063] Through the mechanism of "multi-dimensional highlight extraction + weight optimization sorting", the problem that traditional highlight extraction only relies on text keywords and is disconnected with audience interest is solved. On the one hand, the language large model extracts highlight sentences in combination with preset dimensions such as core selling points of goods and preferential information; on the other hand, audience interaction data and product type templates are introduced to adjust the weight of sentences, so that the highlight sequence contains not only key information of goods, but also audience focus and cognitive logic.

[0064] In the fourth step, the positioning and insertion of product close-up shots: the scene video is divided, labeled and numbered, the visual large model is input to obtain the grid number where the product is located to position the product, the close-up shot parameters are dynamically adjusted according to the type of the product, and the close-up shot is inserted into the corresponding interval of the highlight sequence through fade-in and fade-out transition. Specifically, the video frame is divided into a 50x50 pixel grid picture, the center of each grid is marked with a black-edged red dot (diameter 5 pixels) and numbered, forming a labeled frame image, the labeled frame image is input into the visual large model, and the content of the highlight sentence is combined, such as "look at the texture of the zipper", the grid number where the product is located is output, such as "number 23", the close-up shot parameters are dynamically adjusted according to the type of the product: small goods (such as jewelry, lipstick): magnified to 60%-70% of the picture ratio, 2-3 seconds long, highlighting details; large goods (such as furniture, home appliances): magnified to 40%-50% of the picture ratio, 3-5 seconds long, taking into account the whole and the part, the generated close-up shot picture is inserted into the corresponding time interval of the highlight sentence, such as the sentence "zipper texture" corresponding to 00:08:10-00:08:15, the close-up shot is inserted at 00:08:12-00:08:14, and the picture is smoothly transitioned through fade-in and fade-out transition.

[0065] Through the "grid labeling positioning + dynamic parameter adaptation" technology, the problem of inaccurate description of the position of the goods by the visual large model and the incoordination of the close-up shot and the picture is solved. The video frame is divided into a 50x50 pixel grid and labeled with a red dot with a serial number, the visual large model only needs to output the serial number to accurately position the product, avoiding the "illusion phenomenon"; at the same time, the close-up shot parameters are dynamically adjusted according to the type of the product, and the close-up shot and the main picture are smoothly transitioned through fade-in and fade-out transition. This process ensures that the close-up shot of the product highlights the details without damaging the integrity of the picture, so that the audience can clearly capture the key features of the product, and the visual expressiveness is comparable to manual fine editing, but the efficiency is greatly improved.

[0066] Step 5: Intelligent keyword extraction and material matching: Extract keywords and sort them by frequency and importance, generate text descriptions for each material containing details, uses, and emotions, convert keywords and material descriptions into vectors, and match the top-ranked materials by similarity. When high-frequency keywords appear, prioritize matching the corresponding dynamic material at the corresponding time point. Specifically, use a language large model combined with precision prompt words to extract keywords: input the scene text and highlight sentences, set the prompt words to "extract product name, core selling point, discount information, and emotional tendency", sort the extracted keywords by frequency and importance, and pre-process the materials to generate description texts based on "detail description + use + emotional label". For example, the "red 'limited time' label" description is "a red round rectangle label used to highlight limited-time offers with an emotional tendency of 'urgency'". Convert keywords and material text descriptions into vectors, calculate the matching degree by cosine similarity, and select the top three materials. If the keyword "discount" appears more than 5 times, prioritize matching the dynamic sticker with a countdown animation. Insert the matched materials into the corresponding positions in the timeline, such as adding a sticker to the upper right corner of the screen when the "discount" keyword appears.

[0067] Through the "language large model keyword extraction + material text description vector matching" mechanism, the limitations of traditional TF-IDF algorithm keyword extraction and material matching interference are overcome. After the language large model extracts precise keywords such as product name and core selling point combined with fine-tuning prompt words, the materials are converted into vectors through "detail description + use + emotional label" text processing, and then matched by cosine similarity, reducing the interference data of direct image vector matching, and improving the material matching hit rate by more than 40%. At the same time, when high-frequency keywords appear, the preset stickers are prioritized for matching, making the packaging materials highly compatible with the scene content, enhancing the video's information transmission efficiency and visual appeal.

[0068] In the sixth step, the video is integrated, packaged and output adaptively: the main track, the close-up track and the material track content are integrated, the style and length are adjusted according to the output platform, the picture occlusion is detected and adjusted before output, and finally the video and metadata are output. Specifically, the close-up track is superimposed on the main track, and the material track is inserted according to the timestamp, and the preset style parameters are adjusted according to the output platform, such as TikTok and Taobao. For example, TikTok adopts fast-paced transition length ≤0.5 seconds and bright color matching; Taobao adopts clear subtitle font size ≥24pt and low saturation sticker. Length adaptation: if the recommended length of the platform is 15 seconds, the top 5 highlight sentences are retained; if it is 1 minute, the top 10 highlight sentences are retained to ensure the integrity of the core information. Before output, the visual large model is used to detect whether there is occlusion in the picture, such as a sticker occluding a product. If there is occlusion, the material position is automatically adjusted, and the final video and metadata, such as scene title and keywords, are output. Among them, the close-up track represents the close-up picture of the product; the main track represents the video segment picture corresponding to the highlight sentence; and the material track represents the matched material.

[0069] Through the mechanism of "multi-track integration + platform adaptive adjustment + pre-output verification", the problem of single output format and mismatch with platform characteristics in the existing editing method is solved. Multi-track integration ensures the layered cooperation of highlight segments, product close-ups and packaging materials, avoiding picture confusion; the style parameters are adjusted according to the output platform to make the video meet the platform transmission characteristics; and the picture occlusion is detected by the visual large model before output and automatically adjusted to ensure the integrity and professionalism of the final video. This process improves the secondary transmission conversion rate of the video on different platforms by more than 30%, and realizes rapid output within 10 minutes after the live broadcast ends, greatly improving the editing efficiency.

[0070] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A video clip method based on multi-modal large model collaboration, characterized in that, Specifically comprising the following steps: S1, video preprocessing and intelligent segmentation: preprocessing the original video, cutting the video into small blocks, ensuring that the end of each small block is a complete sentence, outputting the timestamps of each block, audio text segments and storing them as structured data; S2, multi-modal scene separation and verification: inputting the audio text segments and their timestamps into the language large model, initially dividing the scenes according to the rules and reconstructing the spirit diagram, inputting the visual large model to verify whether the goods have been switched, if the goods have not been switched, reinitializing the division, verifying again, and thus dynamically adjusting the cycle, and finally outputting the scene time, title and synopsis; S3, product highlight sentence extraction and sorting: inputting the scene text into the language large model, extracting highlight sentences in combination with multi-dimensional highlight information, introducing audience interaction data and product type preset templates to adjust the weight of the sentences, and sorting the highlight segments according to the weight to form a highlight timeline; S4, product visual close-up positioning and insertion: dividing, marking and numbering the scene video, inputting the visual large model to obtain the grid number where the product is located to position the product, dynamically adjusting the close-up parameters according to the product type, and inserting the close-up into the corresponding interval of the highlight sequence through fade-in and fade-out transitions; S5, intelligent keyword extraction and material matching: extracting keywords and sorting them according to frequency and importance, generating text descriptions containing details, uses and emotions for each material, converting the keywords and material descriptions into vectors, matching the materials with high similarity, and presetting the dynamic materials to be inserted at the corresponding time points when the keywords appear frequently; S6, video integration, packaging and adaptive output: integrating the main track, close-up track and material track contents, adjusting the style and length according to the output platform, detecting the picture obstruction before outputting, and finally outputting the video and metadata.

2. The video clip method based on multi-modal large model collaboration according to claim 1, characterized in that: The step S1 specifically comprises the following steps: S11, audio feature extraction: separating the audio from the original video, extracting the sound intensity sequence and speech pause features; S12, semantic integrity judgment: converting the audio into text by calling the speech recognition model, analyzing the sentence boundaries of the text by the language large model, and detecting the time points corresponding to the punctuation including the period and exclamation point; S13, block rule execution: combining the sound intensity, low-intensity segments are preferred as breakpoints and sentence end times, cutting the video into small blocks with a preset time range, ensuring that the end of each small block is a complete sentence; S14, block result storage: outputting the start and end timestamps of each block, the corresponding audio text segments, and storing them as structured data.

3. The video clip method based on multi-modal large model collaboration according to claim 1, characterized in that: The step S2 specifically comprises the following steps: S21, input each segmented text of the segmented text into a language large model, perform scene division based on preset rules, and have the following strategies for the segmentation result. When no goods are segmented or only one good is segmented, it is considered that the good explanation has not ended, and the next subtitle data will be spliced; when two goods are segmented and the explanation of the second good lasts less than two minutes, it is considered that it has not ended, and delay for two minutes to ensure stable segmentation; when two goods are segmented and the explanation of the second good lasts more than two minutes, it is considered that the explanation of the first good has ended, and the first segmentation result is output, and the subtitle of the second good is spliced; when multiple segmentation results are obtained, only the last two results are reserved, and the rest are output; when multiple short segmentation results less than two minutes are obtained, only the last two results are reserved, and the rest are combined into one result and named as "complex scene"; S22, extract frame images at a frequency of one frame per second from the video within a certain preset time before and after the segmentation point of the preliminary segmentation result, splice them into a sprite with a horizontal arrangement in chronological order, and set the frame spacing to 10 pixels; S23, input the sprite into a visual large model to identify the switching features of the goods, and if the visual confirms the switching of the goods, the initial segmentation result is reserved; S24, if the switching is not confirmed, extend the segmentation point backward according to the continuity of the slow motion, and re-execute steps S21-S23 until the visual confirms the accuracy of the segmentation; S25, output the scene text including the start and end time, scene title and content summary of each scene.

4. The video clip method based on multi-modal large model collaboration according to claim 1, characterized in that: The step S3 specifically includes the following steps: S31, input the scene text into a language large model, extract highlight sentences in combination with preset highlight dimensions, and output the start and end time and content of the sentences; S32, introduce auxiliary data to optimize the highlight sorting, including combining the keywords of the live broadcast and the peak time of likes, increasing the audience interaction weight of the sentences referring to the corresponding content, and presetting the commodity type weight of the sorting template according to the commodity type; S33, sort the highlight sentences according to the template order based on the weight calculation result, and form a highlight segment basic timeline.

5. The video clip method based on multi-modal large model collaboration according to claim 1, characterized in that: When the step S4 inserts the close-up positioning of the goods, the video frame is divided into a 50x50 pixel grid picture, the center of each grid is marked with a black outlined red dot and labeled with a serial number, a marked frame image is formed, the marked frame image is input into a visual large model, the serial number of the grid where the goods are located is output in combination with the content of the highlight sentences, the close-up parameters are dynamically adjusted according to the type of the goods, the generated close-up picture of the goods is inserted into the time interval of the corresponding highlight sentence, and the picture is smoothly transitioned through fade-in and fade-out transition.

6. The video clip method based on multi-modal large model collaboration according to claim 1, characterized in that: The step S5 adopts a combination of a language large model and precision prompt words to extract keywords when performing intelligent keyword extraction and material matching: input scene text and highlight sentences, the prompt words are set as "extract product name, core buying point, preferential information, emotional tendency", the extracted keywords are sorted according to the frequency of occurrence and the preset importance, the material is preprocessed to generate a description text according to "detail description + use + emotional label", the keywords and the material text description are converted into vectors, the matching degree is calculated through cosine similarity, the top three materials in the matching degree ranking are selected, if the preset keyword frequency of occurrence is greater than the preset frequency, the dynamic sticker of the countdown animation is preferentially matched, and the matched material is inserted into the corresponding position according to the time line.

7. The video clip method based on multi-modal large model collaboration according to claim 1, characterized in that: In the step S6, when performing video integration packaging and adaptive output, the close-up track and the main track are superimposed, and then the material track is inserted according to the time stamp, the preset style parameters and the time length parameters of the output platform, the picture is detected through a visual large model before output to determine whether there is an occlusion, if there is an occlusion, the material position is automatically adjusted, and finally the output video and metadata are obtained; wherein the close-up track represents a product close-up picture; the main track represents a video segmented picture corresponding to the highlight sentence; and the material track represents the matched material.

8. A video clip system based on multi-modal large model collaboration, characterized in that: The system is used to realize the video clipping method based on multi-modal large model cooperation according to any one of claims 1-7, and comprises a video preprocessing and intelligent blocking module, a multi-modal scene separation and verification module, a product highlight sentence extraction and sorting module, a product visual close-up positioning and insertion module, an intelligent keyword extraction and material matching module, and a video integration packaging and adaptive output module. The video preprocessing and intelligent blocking module is used for preprocessing the original video, cutting the video into small blocks, ensuring that the end of each small block is a complete sentence, outputting the time stamp, audio text segment and storing the data in a structured form. The multi-modal scene separation and verification module is used for inputting each block into a language large model, initially separating scenes according to rules and then reconstructing a magic graph, inputting a visual large model to verify whether the product is switched, and finally outputting scene time, title and brief introduction. The product highlight sentence extraction and sorting module is used for inputting scene text into a language large model, extracting highlight sentences in combination with multi-dimensional highlight information, introducing audience interaction data and product type preset templates to adjust the weight of the sentences, and sorting the highlight segments according to the weight to form a highlight segment timeline. The product visual close-up positioning and insertion module is used for dividing, marking and numbering the scene video, inputting the visual large model to obtain the grid number of the product to position the product, dynamically adjusting the close-up parameters according to the product type, and inserting the close-up into the corresponding interval of the highlight sequence through fade-in and fade-out transition. The intelligent keyword extraction and material matching module is used for extracting keywords and sorting them according to the frequency and importance, generating a text description containing details, use and emotion for each material, converting the keywords and material descriptions into vectors, selecting the materials with high matching similarity, and preferentially matching the corresponding dynamic material and inserting it into the corresponding time point when the preset keyword appears frequently. Video integration package and adaptive output module: for integrating main track, close-up track, material track content, adjusting style and length according to output platform, detecting picture occlusion and adjusting before output, finally outputting video and metadata.

Images