Video processing method and device based on large model and ASR

Abstract

The invention discloses a video processing method and device based on a large model and ASR. The method comprises the following steps: receiving and storing uploaded video data; inputting the video data into an ASR model for voice recognition to obtain subtitle information with time nodes, and performing associative storage on the subtitle information and the video data; selecting and linking a big language model according to the configuration instruction, submitting subtitle information with time nodes to the big language model, generating editing point information, and performing associative storage on the editing point information and the video data; and cutting the video data according to the editing point information to generate a plurality of segmented sub-videos, and carrying out associative storage on the segmented sub-videos and the video data. According to the invention, efficient editing and management of video resources are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia intelligent processing, and in particular to a video processing method and device based on a large model and ASR. Background Art

[0002] Currently, with the explosive growth of video content, users' demand for video content is becoming more and more diversified, including but not limited to education, entertainment, news, social networking and other aspects.

[0003] Users have an increasing demand for video editing. However, traditional video editing requires professionals to manually determine the editing nodes of the video in order to process the video in segments. This is often inefficient, lacks intelligence, and is prone to errors. High-quality video editing usually requires professional software and tools, which has a high learning curve for non-professionals. In some application scenarios, such as live streaming, users want video content to be generated and shared quickly, which places higher requirements on processing speed.

[0004] Therefore, how to provide a video processing method and device for realizing efficient editing and management of video resources is a technical problem to be solved urgently by those skilled in the art. Summary of the invention

[0005] In view of the above research status and existing problems, the present invention provides an integrated and intelligent video processing device and method, which realizes efficient editing and management of video resources by means of the latest large model technology and ASR technology.

[0006] The present invention first provides a video processing method based on a large model and ASR, comprising the following steps:

[0007] S1: Receive and store uploaded video data;

[0008] S2: Input the video data into the ASR model for speech recognition, obtain subtitle information with time nodes, and store it in association with the video data;

[0009] S3: selecting and linking a large language model according to the configuration instruction, submitting the subtitle information with time nodes to the large language model, generating clipping point information, and storing it in association with the video data;

[0010] S4: cutting the video data according to the clipping point information to generate a plurality of segmented sub-videos, and storing them in association with the video data.

[0011] Preferably, the S1 further comprises receiving file information related to the video data, wherein the file information and the subtitle information with time nodes are submitted together to the language macro model for generating editing point information.

[0012] Preferably, the subtitle information with time nodes includes the text content of the current sentence, and the start time and end time of the current sentence appearing in the video data.

[0013] Preferably, the configuration instruction in S3 includes one or more of the following configuration items: a selected large language model, the number of clipping points, and the name requirement of the clipped sub-video.

[0014] Preferably, S3 comprises the following steps:

[0015] Splicing the subtitle information with time nodes into a json type data set;

[0016] The json type data set and the configuration items in the configuration instruction together form the prompt of the large language model;

[0017] The large language model corresponding to the prompt link is used to generate clipping point information.

[0018] Preferably, S3 comprises the following steps:

[0019] Convert the input data of multiple different large language model service interfaces into a unified data format, and convert the output data of multiple different large language model service interfaces into a unified data format;

[0020] Each large language model service is mapped to a unique enumeration type identifier, and the large language model is switched by modifying the enumeration value of the large language model service in the configuration instruction.

[0021] Preferably, the step S3 further comprises: regenerating new clipping point information after editing the clipping point information, wherein the new clipping point information is used to execute the step S4.

[0022] Preferably, the method further includes the step of setting and updating video recording information, wherein the video recording information is used to record the video processing status:

[0023] After the uploaded video data is stored, the first video recording information is automatically set;

[0024] After the subtitle information with the time node is associated with the video data and stored, the first video recording information is automatically updated to the second video recording information;

[0025] After the clipping point information is associated with the video data and stored, the second video recording information is automatically updated to the third video recording information;

[0026] After the plurality of segmented sub-videos are associated and stored with the video data, the third video recording information is automatically updated to the fourth video recording information;

[0027] Identify the setting and updating actions of the video recording information and automatically execute S2-S4 in sequence.

[0028] The present invention further provides a video processing device based on a large model and ASR, comprising: a video library management device, an ASR device, an LLM gateway service device, a video editing device and a storage device; wherein,

[0029] The video library management device is used to receive uploaded video data and provide an entry for processing the video data;

[0030] The ASR device is used to input the video data into an ASR model for speech recognition to obtain subtitle information with time nodes;

[0031] The LLM gateway service device is used to select and link a large language model according to a configuration instruction, submit the subtitle information with time nodes to the large language model, and generate clip point information;

[0032] The video editing device is used to cut the video data according to the editing point information to generate a plurality of segmented sub-videos;

[0033] The storage device is used to store the video data, and store the subtitle information with time nodes, the editing point information and the plurality of segmented sub-videos in association with the video data.

[0034] Preferably, the LLM gateway service device is built-in and docked with multiple large language models, and the LLM gateway service device includes a data conversion unit and a service mapping unit:

[0035] The data conversion unit is used to convert input data of multiple different large language model service interfaces into a unified data format, and to convert output data of multiple different large language model service interfaces into a unified data format;

[0036] The service mapping unit is used to map each large language model service to a unique enumeration type identifier, and implement the switching of the large language model by modifying the enumeration value of the large language model service in the configuration instruction.

[0037] Compared with the prior art, it has the following beneficial effects:

[0038] 1. Efficiency: Based on LLM and ASR, the video content can be automatically segmented and edited to generate editing points, achieving intelligent breakthroughs in video content understanding, information extraction, and editing decisions, breaking the limitations of traditional video editing tools that rely solely on manual operations and improving efficiency.

[0039] 2 Flexibility: Through the design of the LLM gateway service device, the present invention has good scalability and flexibility, can adapt to the evolving AI technology needs, and timely access and switch new large model services

[0040] 3. Ease of use: It provides one-click generation and processing operations. You only need to upload the video to perform operations such as video analysis and editing.

[0041] The video processing method and device based on a large model and ASR provided by the present invention realize efficient management and accurate editing of video resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0043] Figure 1 A schematic diagram of a video processing method based on a large model and ASR provided in an embodiment of the present invention;

[0044] Figure 2 A flowchart of the video processing method based on a large model and ASR provided by an embodiment of the present invention;

[0045] Figure 3 A structural block diagram of a video processing device based on a large model and ASR provided in an embodiment of the present invention;

[0046] Figure 4 This is a structural block diagram of the LLM gateway service device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The present invention provides a solution for solving the problem of long loading time caused by excessive font file size, while seeking a convenient and efficient method for optimizing font loading that both ensures the beauty of the interface and improves the user experience.

[0049] The first aspect of the present invention discloses a video processing method based on a large model and ASR, such as Figure 1-2 As shown, the following steps are included: S1: receiving and storing uploaded video data; the video format supports MP4 and WAV formats. S2: inputting the video data into the ASR model (automatic speech recognition model) for speech recognition, obtaining subtitle information with time nodes, and storing it in association with the video data; S3: selecting and linking the large language model according to the configuration instructions, submitting the subtitle information with time nodes to the large language model, generating clipping point information, and storing it in association with the video data; S4: cutting the video data according to the clipping point information, generating several segmented sub-videos, and storing them in association with the video data.

[0050] In one embodiment, S1 also includes receiving file information related to video data. The file information and subtitle information with time nodes are submitted together to the language model to generate editing point information. The file format supports pptx, pdf, word, etc. The file information includes but is not limited to video information, video document information, and video outline information. The document information is used to supplement the content of the video, and the outline information is a summary of the content of the video.

[0051] In one embodiment, the subtitle information with time nodes includes the text content of the current sentence, and the start time and end time of the current sentence appearing in the video data.

[0052] In one embodiment, the configuration instructions in S3 include one or more of the following configuration items: the selected large language model, the number of clipping points (i.e., how many segmented micro-videos a complete video needs to be clipped into), and the name requirements of the sub-video after clipping (including the name length range of the clipping points, i.e., the length of each segmented micro-video name).

[0053] In one embodiment, S3 includes the following steps:

[0054] Splice the subtitle information with time nodes into a json type data set;

[0055] Combine the JSON data set and the configuration items in the configuration instructions to form the prompt of the large language model;

[0056] According to the prompt link corresponding to the large language model, and generate clip point information.

[0057] In one embodiment, S3 includes the following steps:

[0058] Convert the input data of multiple different large language model service interfaces into a unified data format, and convert the output data of multiple different large language model service interfaces into a unified data format;

[0059] Each large language model service is mapped to a unique enumeration type identifier, and the large language model is switched by modifying the enumeration value of the large language model service in the configuration instruction.

[0060] This embodiment is illustrated by an example: first, the subtitle information corresponding to the video data is queried. If the subtitle information of the video does not exist, the video is prompted to perform a voice recognition operation first; if it exists, the queried subtitle information is spliced ​​into a json type collection, such as: [{"startTime":0.0,"video subtitle 1","endTime":"3.1"},{"startTime":4.5,"video subtitle 2",

[0061] "endTime":"10.2",...]. The json type data of the subtitle information, the number of clip points and the clip points together form the prompt of the model, for example: "The content in [] is a subtitle document in a video. Please divide it by knowledge points to segment the video. Please control the number of segmentation points to 3 to 5 (here is the number of clip points). The text summary of each paragraph should be controlled within 10 words (here is the length of the clip point), and the segmentation time period and the knowledge description in the corresponding time period are given. The output format is as follows [(0,15,'summary content 1'), (15,203,'summary content 2')]." Based on the current prompt, and after selecting the corresponding model service and matching its corresponding enumeration value, the LLM gateway service will be called for processing. When the large model returns the result, you can preview and edit the returned result. The data format required to be returned in the prompt is the same as "[(0,15,'summary content 1'),(15,203,'summary content 2')]". The content in each () is a clip point information, and a clip point information includes the start time, end time, and clip point name. The first number in () represents the start time of the clip point in seconds, the second number represents the end time of the clip point in seconds, and the third item in the brackets represents the content summary within this time range, that is, the clip point name, which represents the name of the clipped segmented video.

[0062] In this embodiment, if video file information, such as outline information, is uploaded at the same time as the video data is uploaded, relevant clipping point information can also be output based on the video file information. The execution steps are similar to generating clipping points through subtitle information, except for the generation of prompts. The video file information can be put into prompts.

[0063] In one embodiment, S3 further includes: regenerating new clipping point information after editing the clipping point information, and the new clipping point information is used to execute the step of S4.

[0064] In this embodiment, if you are satisfied with the effect of the editing point information, you can click Save to store the editing point information of the video and associate it with the original video.

[0065] If you are not satisfied with the result, you can manually fine-tune the generated result, including editing the start time, end time, name of the clip point, or merging or deleting some clip points. If you do not want to fine-tune, you can also readjust the configuration items to regenerate the clip point information.

[0066] In one embodiment, the specific execution process of S4 is as follows: when the video performs an editing operation, the editing point information of the video is first queried and obtained. The editing device will cut the video according to the start time and end time of each editing point. The name of each sub-video generated is the name corresponding to the name taken from the editing point. The output sub-video will be uploaded to the corresponding file storage device, and the file address information after uploading will be recorded and associated with the original video data.

[0067] In one embodiment, the step of setting and updating video recording information is also included, where the video recording information is used to record the video processing status:

[0068] After the uploaded video data is stored, the first video recording information is automatically set;

[0069] After the subtitle information with the time node is associated with the video data and stored, the first video recording information is automatically updated to the second video recording information;

[0070] After the editing point information is associated with the video data and stored, the second video recording information is automatically updated to the third video recording information;

[0071] After the plurality of segmented sub-videos are associated and stored with the video data, the third video recording information is automatically updated to the fourth video recording information;

[0072] Identify the setting and updating actions of the video recording information and automatically execute S2-S4 in sequence.

[0073] In this embodiment, after S1 is uploaded, the corresponding video recording information will be generated, and the video processing status will be pending; after S2 has the subtitle information with the time node associated with the video data and stored, the video recording information will be automatically updated to the recognized status; after S3 the editing point information is associated with the video data and stored, the video recording information will be automatically updated to the generated editing point status; after S4 several segmented sub-videos are associated with the video data and stored, the video recording information will be automatically updated to the edited status.

[0074] If you do not want to upload, identify, and edit the video one by one, you can use the one-click processing operation to combine the video uploading, identification, editing point generation, and video editing processing operations into a complete operation by identifying the settings and updating actions of the video recording information. The entire process is automatically processed, and users can edit a complete video without feeling it.

[0075] The second aspect of the present invention provides a video processing device based on a large model and ASR according to the first aspect, such as Figure 3-4 As shown, it includes: a video library management device, an ASR device, an LLM gateway service device, a video editing device and a storage device; wherein,

[0076] A video library management device, used to receive uploaded video data and provide an entry for processing the video data;

[0077] An ASR device is used to input video data into an ASR model for speech recognition to obtain subtitle information with time nodes;

[0078] The LLM gateway service device is used to select and link the large language model according to the configuration instruction, submit the subtitle information with the time node to the large language model, and generate the editing point information;

[0079] A video editing device, used for cutting the video data according to the editing point information to generate a plurality of segmented sub-videos;

[0080] The storage device is used to store video data and associate subtitle information with time nodes, editing point information and a plurality of segmented sub-videos with the video data for storage.

[0081] In one embodiment, the video library management device has an integrated video data operation portal, including: video uploading, video viewing, voice recognition, generating clipping points, video clipping, etc. The video library management device has a video data record list, in which the corresponding sub-videos generated by clipping can be viewed.

[0082] In one embodiment, the storage device includes a file storage device and a database, wherein the file storage device is mainly used to store file information related to video data, including but not limited to video information, video document information, video outline information, and video subtitle information. The database is used to store video data, and subtitle information with time nodes, editing point information, and a plurality of segmented sub-videos stored in association with the video data.

[0083] In one embodiment, the LLM gateway service device is built-in to connect to multiple large language models. The LLM gateway service device includes a data conversion unit and a service mapping unit:

[0084] A data conversion unit, used to convert input data of multiple different large language model service interfaces into a unified data format, and to convert output data of multiple different large language model service interfaces into a unified data format;

[0085] The service mapping unit is used to map each large language model service to a unique enumeration type identifier, and to switch the large language model by modifying the enumeration value of the large language model service in the configuration instruction. When switching is required, you only need to adjust the enumeration value corresponding to the model service to achieve fast switching.

[0086] In this embodiment, the LLM gateway service device has built-in connection with some mainstream large language model service capabilities, such as Tongyi Qianwen, Wenxin Yiyan, etc. Through code adaptation, the input and output parameters of different model service interfaces are converted into a unified format.

[0087] In this embodiment, an interface call recording unit is also included, which is used to record the calling status of the interface when switching or calling the large language model.

[0088] In this embodiment, there is also an authority determination unit for determining whether the operator has the authority to call the current large language model or switch the large language model, thereby providing a technical basis for the switching and authentication of model services.

[0089] In one embodiment, the input interface of the configuration instruction is set on the page to receive the selection instruction of the configuration item. After the configuration of the configuration item is completed, click OK and the parameters will be packaged according to the selected configuration item.

[0090] The second aspect of the embodiment of the present invention can be used to execute all the steps of the first aspect of the embodiment.

[0091] The specific implementation method of the present invention is described below through a teaching video case.

[0092] Usually a classroom video will contain relatively rich knowledge points, and the video length is 45 minutes. If the video can be divided and edited according to knowledge points to form several micro-videos, it will be convenient for students with different foundations to conduct targeted review and consolidation after class, and fill in the gaps. At the same time, it can also form the school's own video knowledge base.

[0093] S100: In the video library management device, upload the classroom video that needs to be edited. After the upload is completed, the corresponding video record information will be generated and the status will be pending.

[0094] S200: After the video is uploaded successfully, select the corresponding video data on the video list page and click the "Voice Recognition" operation. At this time, the ASR device will perform voice recognition on the video and output the voice-to-text information of the entire video and subtitle information with time nodes. The recognized information will be stored in association with the video and the video record will be updated to the recognized state.

[0095] S300: When the video status changes to the recognized status, you can proceed to the next step of generating clip points. Click the "Generate Clip Points" operation, and a page will pop up to select configuration items. First, select the model that has been built-in and connected to the LLM gateway. In this case, select Tongyi Qianwen in the list below, and then choose to generate 5 clip points, and the name length of each clip point is controlled to be within the range of 10. After the selection is completed, click Confirm to generate. At this time, the combined prompt will be submitted to the LLM gateway service device, and the corresponding large model recognition processing will automatically generate the relevant information of the video clip points, and the generated results will be displayed on the page. If you do not adjust the generated results, you can click to save the clip point information. At this time, the video is updated to the state of generated clip points.

[0096] If you need to adjust, you can edit the start time, end time, name of the clip point, or merge or delete some clip points. You can also re-edit the configuration items and regenerate the clip point information based on the latest configuration items.

[0097] S400: When the video changes to the state of generated editing points, you can edit the video. Click the "Edit" operation. The video editing device will cut the video according to the editing point information and generate segmented micro-videos. The micro-videos will be associated with the original video and stored. After the editing is completed, the video status will be updated to the edited status. At this time, you can view the video generated by the editing in the original video record.

[0098] The above is a detailed introduction to a video processing method and device based on a large model and ASR provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0099] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A video processing method based on a large model and ASR, characterized in that: The steps include: S1: Receive and store uploaded video data; S2: Input the video data into the ASR model for speech recognition, obtain subtitle information with time nodes, and store it in association with the video data; S3: selecting and linking a large language model according to the configuration instruction, submitting the subtitle information with time nodes to the large language model, generating clipping point information, and storing it in association with the video data; S4: cutting the video data according to the clipping point information to generate a plurality of segmented sub-videos, and storing them in association with the video data.

2. The video processing method based on a large model and ASR according to claim 1, characterized in that: The S1 also includes receiving file information related to the video data, and submitting the file information and the subtitle information with time nodes together to the language big model for generating editing point information.

3. The video processing method based on large model and ASR according to claim 1, characterized in that: The subtitle information with time nodes includes the text content of the current sentence, and the start time and end time of the current sentence appearing in the video data.

4. The video processing method based on large model and ASR according to claim 1, characterized in that: The configuration instructions in S3 include one or more of the following configuration items: a selected large language model, the number of clipping points, and the name requirements of the sub-video after clipping.

5. The video processing method based on large model and ASR according to claim 1, characterized in that: The S3 comprises the following steps: Splicing the subtitle information with time nodes into a data set of JSON type; The json type data set and the configuration items in the configuration instruction together form the prompt of the large language model; The large language model corresponding to the prompt link is used to generate clipping point information.

6. The video processing method based on large model and ASR according to claim 1, characterized in that: The S3 comprises the following steps: Convert the input data of multiple different large language model service interfaces into a unified data format, and convert the output data of multiple different large language model service interfaces into a unified data format; Each large language model service is mapped to a unique enumeration type identifier, and the large language model is switched by modifying the enumeration value of the large language model service in the configuration instruction.

7. The video processing method based on large model and ASR according to claim 1, characterized in that: The step S3 further includes: regenerating new clipping point information after editing the clipping point information, wherein the new clipping point information is used to execute the step S4.

8. The video processing method based on large model and ASR according to claim 1, characterized in that: The step of setting and updating video recording information is also included, wherein the video recording information is used to record the video processing status: After the uploaded video data is stored, the first video recording information is automatically set; After the subtitle information with the time node is associated with the video data and stored, the first video recording information is automatically updated to the second video recording information; After the clipping point information is associated with the video data and stored, the second video recording information is automatically updated to the third video recording information; After the plurality of segmented sub-videos are associated and stored with the video data, the third video recording information is automatically updated to the fourth video recording information; Identify the setting and updating actions of the video recording information and automatically execute S2-S4 in sequence.

9. A video processing device based on a large model and ASR, characterized in that: include: Video library management device, ASR device, LLM gateway service device, video editing device and storage device; wherein, The video library management device is used to receive uploaded video data and provide an entry for processing the video data; The ASR device is used to input the video data into an ASR model for speech recognition to obtain subtitle information with time nodes; The LLM gateway service device is used to select and link a large language model according to a configuration instruction, submit the subtitle information with time nodes to the large language model, and generate clip point information; The video editing device is used to cut the video data according to the editing point information to generate a plurality of segmented sub-videos; The storage device is used to store the video data, and store the subtitle information with time nodes, the editing point information and the plurality of segmented sub-videos in association with the video data.

10. The video processing device based on a large model and ASR according to claim 7, characterized in that: The LLM gateway service device is built-in and docked with multiple large language models. The LLM gateway service device includes a data conversion unit and a service mapping unit: The data conversion unit is used to convert input data of multiple different large language model service interfaces into a unified data format, and to convert output data of multiple different large language model service interfaces into a unified data format; The service mapping unit is used to map each large language model service to a unique enumeration type identifier, and implement the switching of the large language model by modifying the enumeration value of the large language model service in the configuration instruction.

Images