Video title generation method and device

Abstract

The embodiment of the invention discloses a video title generation method and device. The method comprises the steps that multi-modal data in a video to be processed are acquired, and the multi-modal data comprise video frames and audio data; and determining a video title prompt keyword according to the multi-modal data, generating video title prompt information according to the video title prompt keyword, inputting the video title prompt information and the multi-modal data into a pre-trained multi-modal large language model, and generating a video title corresponding to the to-be-processed video. Therefore, the multi-modal big language model can be combined with the multi-modal data of the video content to automatically generate the video title, so that the generated video title is highly matched with the video content, and the generated video title can accurately reflect the video content and attract a user to click, thereby improving the production efficiency and the distribution effect of the video manuscript, and improving the user experience. The user experience is improved, and the manpower and material resource cost is saved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and in particular to a method and device for generating a video title. Background Art

[0002] With the rapid development of the Internet and the popularity of video platforms, video has become a mainstream way of mass consumption. Ordinary users can submit video manuscripts through various video platforms. As an important basis for video content distribution, video titles play a vital role in the click-through rate and viewing rate of videos. The quality of video titles directly affects users' first impression of the video, and thus affects the dissemination effect of the video.

[0003] The existing technologies mainly generate video titles based on simple keyword matching or based on video content. These methods often rely on single-mode data, which makes the generated video titles have a low fit with the video content and cannot accurately reflect the video content, further reducing their appeal to users. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a video title generation method, apparatus, computing device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of an embodiment of the present application, a method for generating a video title is provided, comprising:

[0006] Acquire multimodal data in the video to be processed, wherein the multimodal data includes: video frames and audio data;

[0007] Determine video title prompt keywords according to the multimodal data, and generate video title prompt information according to the video title prompt keywords;

[0008] The video title prompt information and multimodal data are input into a pre-trained multimodal large language model to generate a video title corresponding to the video to be processed.

[0009] Furthermore, generating video title prompt information according to the video title prompt keywords further includes:

[0010] A main prompt sentence is generated according to the video title prompt keywords and multimodal data, and video title prompt information is generated according to the main prompt sentence and preset prompt sentences.

[0011] Furthermore, the video title hint keywords include: video title style hint keywords and video title content hint keywords;

[0012] Determining the video title prompt keywords based on the multimodal data further includes:

[0013] Determine image labels for video frames;

[0014] Acquire a first keyword that has a mapping relationship with the image tag, and determine the first keyword as a video title style prompt keyword;

[0015] A second keyword is extracted from the image tag, and the second keyword is determined as a video title content prompt keyword.

[0016] Further, the video title hint keywords include: video title content hint keywords;

[0017] Determining the video title prompt keywords based on the multimodal data further includes:

[0018] A third keyword is extracted from the audio data and the video format information, and the third keyword is determined as a video title content prompt keyword.

[0019] Further, after determining the image tag of the video frame, the method further includes:

[0020] Determine the time at which the image label appears in the video to be processed;

[0021] Calculate the target time proportion of the image label based on the appearance time;

[0022] Filter image labels based on target time ratio;

[0023] Acquiring the first keyword that has a mapping relationship with the image tag further includes: acquiring the first keyword that has a mapping relationship with the filtered image tag;

[0024] Extracting the second keyword from the image tag further includes: extracting the second keyword from the filtered image tags.

[0025] Furthermore, calculating the target time ratio of the image label according to the appearance time further includes:

[0026] If there are multiple identical image tags, for any of the multiple identical image tags, the initial time proportion of the image tag is calculated according to the appearance time, and the multiple initial time proportions are added together to obtain the target time proportion of the same image tag.

[0027] Furthermore, the training method of the multimodal large language model includes:

[0028] Obtaining sample videos, video metadata corresponding to the sample videos, and video comment information, and performing data annotation processing on the video metadata to obtain annotation results;

[0029] Extract features from sample videos to obtain corresponding video features and audio features;

[0030] Generate subtitle information based on sample videos, and extract text features from subtitle information and video comment information;

[0031] The model is trained based on video features, audio features, text features and annotation results to obtain a multimodal large language model.

[0032] According to another aspect of an embodiment of the present application, a video title generating device is provided, including:

[0033] An acquisition module, adapted to acquire multimodal data in a video to be processed, wherein the multimodal data includes: video frames and audio data;

[0034] A determination module, adapted to determine video title prompt keywords according to the multimodal data, and generate video title prompt information according to the video title prompt keywords;

[0035] The generation module is suitable for inputting video title prompt information and multimodal data into a pre-trained multimodal large language model to generate a video title corresponding to the video to be processed.

[0036] According to another aspect of the embodiment of the present application, there is provided a computing device, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0037] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the above-mentioned video title generation method.

[0038] According to another aspect of the embodiments of the present application, a computer storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned video title generation method.

[0039] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned video title generation method.

[0040] According to the video title generation method and device provided in the embodiment of the present application, first, the multimodal data in the video to be processed is obtained, then, the video title prompt keywords are determined according to the multimodal data, and the video title prompt information is generated according to the video title prompt keywords. Finally, the video title prompt information and the multimodal data are input into the pre-trained multimodal large language model to generate the video title corresponding to the video to be processed. Thus, the multimodal large language model can be used to automatically generate the video title in combination with the multimodal data of the video content, so that the generated video title is highly consistent with the video content, and the generated video title can accurately reflect the video content and attract user clicks, thereby improving the production efficiency and distribution effect of the video manuscript, improving the user experience, and saving manpower and material costs.

[0041] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the embodiment of the present application is listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the embodiments of the present application. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0043] Figure 1 A schematic diagram showing a flow chart of a method for generating a video title according to an embodiment of the present application is shown;

[0044] Figure 2 A schematic diagram showing a flow chart of a method for generating a video title according to another embodiment of the present application is shown;

[0045] Figure 3 Flowchart for video title hint information generation;

[0046] Figure 4 This is a schematic diagram of the application of a multimodal large language model;

[0047] Figure 5 This is a training flowchart for a multimodal large language model;

[0048] Figure 6 System architecture diagram for video title generation system;

[0049] Figure 7 Main flow chart for local processing on the client side;

[0050] Figure 8It is the state transition flowchart of the smart title task;

[0051] Fig. 9 A structural block diagram of a video title generating device according to an embodiment of the present application is shown;

[0052] Fig.10 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0054] First, the terms involved in one or more embodiments of the present application are explained.

[0055] Multimodal Large Language Models: refers to a model that combines multimodal information such as text, images, and audio for training. It is a technology that combines vision and large language models. This model can perceive inputs of different modalities such as pictures, audio, and text, and learn the context and generate answers in an autoregressive manner based on the given instructions.

[0056] AI prompt word: that is, Prompt. In the AI ​​large model, the role of Prompt is mainly to provide the AI ​​model with the context of input information and parameter information of the input model.

[0057] Figure 1 A schematic diagram of a process of generating a video title according to an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0058] Step S101, obtaining multimodal data in a video to be processed, wherein the multimodal data includes: video frames and audio data.

[0059] Specifically, when a user wants to generate a video title for a video he or she has made, he or she can upload the video to be processed through the upload function provided by the client. The client is an application, including but not limited to mobile applications, desktop applications, and web applications. For example, the client can be a submission application.

[0060] After obtaining the video to be processed, multimodal data can be extracted from the video to be processed, wherein the multimodal data includes: video frames and audio data. For example, a number of video frames can be extracted from the above-mentioned video to be processed according to a first preset rule. For example, a frame can be extracted at fixed time intervals. This extraction method is simple and easy to implement, and ensures that the extracted video frames are evenly distributed; key frame detection can also be performed, for example, detecting motion changes between frames, extracting video frames with larger motion changes, so as to capture important events and actions in the video, and / or detecting changes in content between frames, extracting video frames with larger content changes, so as to capture visually significant changes, and / or detecting changes in scenes, extracting video frames with scene changes, so as to capture scene switching in the video. In addition, it can also be based on other rules, which are not specifically limited here.

[0061] An audio file is extracted from the video to be processed, and audio is cropped and transcoded according to a second preset rule to obtain corresponding audio data. For example, the audio within a specified fixed time period can be cropped, or the silent segment in the audio can be detected to retain only the non-silent part, or the sound outside a specific frequency range can be filtered out to retain only the frequency range of interest, etc., which are not specifically limited here. Transcoding refers to the process of converting one audio format into another audio format. Through transcoding, different sound quality requirements can be met, bandwidth usage can be optimized, and compatibility can be ensured.

[0062] Step S102, determining video title prompt keywords according to the multimodal data, and generating video title prompt information according to the video title prompt keywords.

[0063] After obtaining the multimodal data, the video title hint keywords can be determined based on the multimodal data. For example, keywords can be directly extracted from the multimodal data, or by performing content recognition on the video frame and performing word mapping based on the recognition results. Among them, the video title hint keywords are used to guide the video title output by the multimodal large language model, for example, guiding the style category and guiding the title content. If the determined video title hint keywords are highly integrated with the multimodal data.

[0064] Then, based on the video title hint keywords, video title hint information is generated. The generated video title hint information is composed of several sentences. Since the video title hint information is generated based on the video title hint keywords, the generated video title hint information is highly consistent with the video content.

[0065] Step S103, inputting the video title prompt information and the multimodal data into a pre-trained multimodal large language model to generate a video title corresponding to the video to be processed.

[0066] After generating the video title prompt information, the video title prompt information and multimodal data can be input into a pre-trained multimodal large language model. The multimodal large language model processes multimodal input data such as images and audios, and combines the video title prompt information to generate a video title that is highly consistent with the video content.

[0067] According to the video title generation method provided in the embodiment of the present application, the multimodal data in the video to be processed is obtained, the video title prompt keywords are determined according to the multimodal data, and the video title prompt information is generated according to the video title prompt keywords, and the video title prompt information and the multimodal data are input into the pre-trained multimodal large language model to generate the video title corresponding to the video to be processed. Thus, the multimodal large language model can be used to automatically generate the video title in combination with the multimodal data of the video content, so that the generated video title is highly consistent with the video content, and the generated video title can accurately reflect the video content and attract user clicks, thereby improving the production efficiency and distribution effect of the video manuscript, improving the user experience, and saving manpower and material costs.

[0068] Figure 2 A schematic diagram of a process of generating a video title according to an embodiment of the present application is shown. Figure 2 As shown, the method comprises the following steps:

[0069] Step S201, obtaining multimodal data in a video to be processed, wherein the multimodal data includes: video frames and audio data.

[0070] The specific implementation method of this step is Figure 1 Step S101 in the illustrated embodiment is similar and will not be described in detail here. There are multiple video frames.

[0071] In addition, multimodal data may also include: text data, for example, determining the video format information corresponding to the video to be processed as text data. In addition, the user may also make a corresponding description for the video to be processed, which is referred to as video description information, and use the video description information as text data.

[0072] Step S202: determining the image label of the video frame.

[0073] Specifically, a pre-trained image label recognition model can be used to perform content recognition processing on video frames, thereby generating image labels that can reflect the main content of the video frames. By performing content recognition processing on all video frames, multiple icon labels can be obtained. The image label recognition model is specifically trained based on video frames and corresponding image label annotations.

[0074] After the image tags are determined, video title prompt information can be generated based on the image tags and multimodal data. Figure 3 The flowchart for generating the video title prompt information is as follows: Figure 3 As shown, after image label screening, style mapping based on the screened image labels, keyword extraction, and keyword extraction based on multimodal data, video title prompt information is generated based on the extracted keywords, keywords obtained by style mapping, multimodal data, and output restriction information. The specific implementation process can be referred to steps S203 to S208:

[0075] Since multiple video frames are extracted, after content recognition is performed on the video frames, a large number of image tags may be obtained. In order to effectively improve the efficiency of video title generation and avoid too many image tags affecting the accuracy of the generated video titles, it is necessary to filter the image tags. There are many methods for filtering image tags, for example, filtering image tags according to their frequency of occurrence, for example, counting the frequency of occurrence of each image tag in the entire video to be processed, and filtering image tags according to the frequency of occurrence, for example, filtering image tags whose frequency of occurrence is greater than or equal to a preset frequency threshold, or filtering image tags according to the proportion of time occupied by the image tags.

[0076] Specifically, the method in step S203 to step S205 can be used to filter the image tags:

[0077] Step S203, determining the appearance time of the image tag in the video to be processed.

[0078] Specifically, when obtaining video frames from the video to be processed, the timestamp corresponding to each video frame will be recorded. After performing content recognition processing on the video frames and determining the image tags corresponding to the video frames, the timestamp corresponding to the video frames can be determined as the appearance time of the image tags in the video to be processed.

[0079] Step S204, calculating the target time ratio occupied by the image tag according to the appearance time.

[0080] After determining the appearance time of the image tag in the video to be processed, the target time ratio of the image tag can be calculated according to the appearance time. For example, the appearance duration corresponding to the image tag can be determined by combining the timestamp of the video frame adjacent to the video frame corresponding to the image tag (referring to the extracted previous video frame). For the sake of ease of description, the timestamp of the video frame corresponding to the image tag is recorded as the first timestamp, and the timestamp corresponding to the adjacent video frame is recorded as the second timestamp. The appearance duration = the first timestamp - the second timestamp, and the target time ratio = the appearance duration / the video duration of the video to be processed. If the video frame corresponding to the image tag is the first video frame extracted, then 0 seconds is used as the starting time for calculating the appearance duration.

[0081] Alternatively, the target time ratio occupied by the image label can be determined by the following method: determine the appearance time of the image label in the video to be processed, divide the video to be processed into a preset number of time windows, count the number of appearances of the image label in each window, and divide the total number of appearances of each image label in all windows by the total number of windows to obtain the target time ratio.

[0082] Step S205: filter image labels according to the target time ratio.

[0083] After determining the target time ratio corresponding to the image tag, the image tags can be screened according to the target time ratio. For example, a preset number of image tags with the highest target time ratio, such as 3 or 5 image tags, can be screened out, or image tags with a target time ratio greater than or equal to the preset time ratio can be screened out.

[0084] In an optional implementation, when determining the image label of a video frame, there may be a situation where multiple video frames correspond to the same image label, that is, after content recognition processing, there may be multiple identical image labels. For the situation where there are multiple identical image labels, the following method can be used to calculate the target time ratio corresponding to the image label: for any image label among the multiple identical image labels, the initial time ratio occupied by the image label is calculated according to the appearance time. The specific calculation method is similar to step S204 and will not be repeated here. Through calculation, multiple initial time ratios will be obtained, and then the multiple initial time ratios will be added and summed. The sum result is the target time ratio of the same image label.

[0085] Step S206, obtaining a first keyword that has a mapping relationship with the filtered image tag, and determining the first keyword as a video title style prompt keyword.

[0086] After filtering out the image tags, style mapping can be performed based on the filtered image tags. Specifically, the mapping relationship between different image tags and video title style hint keywords is pre-set in advance and recorded in the style hint keyword list. Each style has a corresponding fixed video title style hint keyword. Therefore, after filtering out the image tags, the style hint keyword list can be queried based on the filtered image tags to obtain the first keyword that has a mapping relationship with the filtered image tags. The first keyword is the video title style hint keyword. The video title style hint keyword can guide the multimodal large language model to generate video titles of different style categories.

[0087] The following is a brief list of video title style hint keywords determined after style mapping of different image tags:

[0088] When the image tags are: game, mobile game, the keywords of the video title style are: strategy, funny and humorous;

[0089] When the image tags are: landscape, natural scenery, and outdoor, the mapped video title style prompt keywords are: poetic beauty, daily record, and VLOG.

[0090] Step S207, extracting a second keyword from the filtered image tags, extracting a third keyword from the audio data and the video format information, and determining the second keyword and the third keyword as video title content prompt keywords.

[0091] Keywords are extracted from the filtered image tags, audio data, and video format information. For example, keywords such as 2K, 4K, and ultra-clear are extracted from video resolutions; keywords such as scenery, sports, and games are extracted from image tags. The extracted keywords are used as video title content prompt keywords to guide the output content of the multimodal large language model.

[0092] Step S208, generating a main prompt sentence according to the video title content prompt keywords, the video title style prompt keywords and the multimodal data, and generating video title prompt information according to the main prompt sentence and the preset prompt sentence.

[0093] After determining the video title content prompt keywords and the video title style prompt keywords, a main prompt sentence can be generated based on the video title content prompt keywords, the video title style prompt keywords and the multimodal data, wherein the multimodal data is used as input information to prompt what type of data is input to the multimodal large language model. The main prompt sentence is a key prompt sentence in the video title prompt information, which generally defines the information on which the multimodal large language model outputs the content, as well as the requirements that need to be met. For example, the corresponding main prompt sentence can be generated in combination with the corresponding prompt sentence framework.

[0094] For example, when the video title style hint keyword is poetic and aesthetic, the following main hint sentence can be generated based on the above video title style hint keyword: Please generate a title with poetic and aesthetic language style, advanced and exquisite wording, ancient Chinese style, and quotations from classics.

[0095] When the video title style hint keyword is warm and healing, the following main hint sentence can be generated based on the above video title style hint keyword: Please generate a title with a warm and cozy atmosphere that makes you feel healed.

[0096] For example, the extracted video title content prompt keywords are: 4K, ultra-clear, and scenery. Then, according to the above video title content prompt keywords, the following main prompt sentence can be generated: Please combine the following keywords: 4K, ultra-clear, scenery, etc. to generate the video title. Specifically, the video title content prompt keywords can be filled into the first preset prompt sentence framework to generate the main prompt sentence corresponding to the video title content prompt keywords. Among them, the first preset prompt sentence framework can be: Please combine the following keywords: XX to generate the video title.

[0097] In addition, the multimodal data input to the multimodal large language model needs to be described in the video title prompt information. For example, the multimodal data includes: video frames and audio data, then the generated main prompt sentence can be: please extract key content from the input video frames and audio data, and generate a video title based on the extracted content.

[0098] Then, the video title prompt information is generated according to the generated main prompt sentence and the preset prompt sentence, wherein the preset prompt sentence may include the following parts:

[0099] Assumptions section:

[0100] Prompt phrases provide contextual information to the model, such as the role the model plays, the professional skills it possesses, the current scenario, and the tasks that need to be performed. Example:

[0101] Assumed role:

[0102] You are a video blogger who has been in the industry for many years and is good at producing video content. Every time you can attract more traffic exposure and user clicks with your novel titles, xxx.

[0103] Assumed Skills:

[0104] You have the following professional skills, which are only examples and do not have any limiting effect:

[0105] Skill 1: Video content extraction

[0106] Accurately capture the core elements of the picture, video or audio, summarize the title, and provide positive, active and healthy content

[0107] Skill 2: Video title creation

[0108] Refer to the output requirements and input information, and create concise and clear titles based on the content extracted from the video. The title needs to extract the core highlights of the content as much as possible, and the voice should be plain and natural without exaggeration to ensure that it conforms to the language habits and preferences of video website users.

[0109] Assumptions and tasks:

[0110] Now that you have produced a high-quality video, you need to generate multiple video titles that fit the video content.

[0111] Output restriction section:

[0112] To ensure the security, accuracy and stability of the output results, the output content is restricted by prompt words. Example:

[0113] The output content must strictly follow the following restrictions, which are only examples and do not have any limiting effect:

[0114] - Output 5 titles

[0115] - Each title must be unique and avoid repeated words

[0116] - Strictly abide by the word limit, the title should not exceed 20 words

[0117] -Only output the final generated title, and prohibit the output of irrelevant content

[0118] - Title language is only in Chinese

[0119] Therefore, combined with the above examples, the following video title prompt information can be generated:

[0120] You are a video blogger who has been in the industry for many years. You are good at producing video content. You can always attract more traffic exposure and user clicks because of the novel titles. You have the following professional skills:

[0121] Skill 1: Video content extraction

[0122] Accurately capture the core elements of pictures, videos or audios, summarize the titles, and provide positive, active and healthy content

[0123] Skill 2: Video title creation

[0124] Refer to the title sentence structure in the reference sentence structure requirements, and create concise and clear titles based on the content extracted from the video. The title needs to extract the core highlights of the content as much as possible, and the language should be plain and natural without exaggeration to ensure that it conforms to the language habits and preferences of video website users.

[0125] Please use video content extraction to extract the core elements from the input images and audio, and combine the following keywords: 4K, ultra-clear, landscape, outdoor, to generate 5 titles with poetic and beautiful language styles, sophisticated and refined words, ancient Chinese style, and quotations from classics. Your output content must strictly follow the following requirements:

[0126] - Only output 5 titles

[0127] - Each title must be unique and avoid repeated words

[0128] - Strictly abide by the word limit, the title should not exceed 20 words

[0129] -Only output the final generated title, and prohibit the output of irrelevant content

[0130] - Title language is only in Chinese

[0131] For different videos to be processed, the generated video title prompt information is different, and the generated video title prompt information is highly integrated with the video content. Therefore, the final generated video title is also different and highly consistent with the video content.

[0132] Step S209: Input the video title prompt information and multimodal data into a pre-trained multimodal large language model to generate and display the video title corresponding to the video to be processed.

[0133] After the video title prompt information is generated, the video title prompt information and multimodal data can be input into a pre-trained multimodal large language model, which processes multimodal input data such as images and audio, and combines the video title prompt information to generate a video title that is highly consistent with the video content. Figure 4 As shown, the application process of the multimodal large language model is shown.

[0134] Among them, the training method of the multimodal large language model includes:

[0135] Obtaining sample videos, video metadata corresponding to the sample videos, and video comment information, and performing data annotation processing on the video metadata to obtain annotation results;

[0136] Extract features from sample videos to obtain corresponding video features and audio features;

[0137] Generate subtitle information based on sample videos, and extract text features from subtitle information and video comment information;

[0138] The model is trained based on video features, audio features, text features and annotation results to obtain a multimodal large language model.

[0139] Figure 5 This is the training flow chart of the multimodal large language model, such as Figure 5 As shown in the figure, the training of multimodal large language model mainly includes the following parts:

[0140] (1) Determination of demand content:

[0141] Objective definition: Clarify the application scenarios of the model. The main application scenario of this application is to generate appropriate titles for users' videos.

[0142] Data modality: determines the data type of the data, which in this application includes video frames, metadata (title, description), user comments, etc.

[0143] Model Sizing

[0144] (2) Data collection and preparation:

[0145] Data collection: Collect video footage and corresponding titles and descriptions

[0146] Data preprocessing: Preprocess the video, including extracting key frames, extracting audio features, generating subtitles, etc. Clean and denoise the text data.

[0147] Data annotation: Annotate data, such as scoring titles, keyword annotation, etc.

[0148] (3) Feature extraction

[0149] Video feature extraction: Extract visual features of video frames using a pre-trained vision model

[0150] Text feature extraction: Extract text features using a pre-trained language model.

[0151] Audio Feature Extraction: Extract audio features using a pre-trained audio model.

[0152] (4) Model training

[0153] Model selection: Based on the required content, select the model to integrate.

[0154] Data segmentation: Divide the data set into training set, validation set and test set

[0155] Supervised learning: Use labeled data to perform supervised training on the model, with the goal of optimizing to produce titles that are close to the actual titles.

[0156] Model tuning: Performance indicators: BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation), two evaluation indicators, are introduced to measure the quality of generated titles.

[0157] The model parameters are continuously adjusted according to the above evaluation results to optimize the model.

[0158] (5) Model deployment

[0159] Model optimization: Prune and quantize the model to facilitate deployment in the production environment

[0160] Model deployment: Deploy the model on the platform cloud server.

[0161] API design: Design and implement an intelligent title recommendation interface for external systems to call the model to generate video titles.

[0162] In addition, a compliance review mechanism will be introduced to ensure that the content generated by the model complies with laws, regulations, and ethical requirements.

[0163] In an optional implementation, after the video title is generated, the generated video title can be recommended to the user, and the user can edit or apply the recommended video title. Multiple video titles can also be recommended to the user, and the user can select a video title from them.

[0164] In an optional implementation, the user can also provide style information of the video title to be generated, and the style information will be provided to the multimodal large language model as text data, so that titles of different types and styles can be recommended according to the user's needs, thereby meeting the user's diverse needs for video distribution effects.

[0165] According to the video title generation method provided in the embodiment of the present application, by screening image tags, the accuracy of the generated video title and the efficiency of video generation can be effectively improved, and the multimodal large language model can be used in combination with the multimodal data of the video content to automatically generate the video title, so that the generated video title is highly consistent with the video content, and the generated video title can both accurately reflect the video content and attract user clicks, thereby improving the production efficiency and distribution effect of video manuscripts, improving user experience, and saving manpower and material costs; based on image tags and multimodal data, video title prompt information is generated, so that the video title prompt information and the user's manuscript content are highly matched, so that the video title generated based on the video title prompt information is highly consistent with the video content.

[0166] The video title generation method provided in the above embodiment can be executed by the client, or by the server, or by the client and the server together. Since it takes a certain amount of time for the user to upload the video and for the client to upload the video to the server, it is preferred that the client and the server jointly execute the method. Specifically, the multimodal data is extracted and the image tags of the video frames are determined locally on the client. In the process of the user uploading the video, it can be achieved while uploading and extracting, and the image tags of the video frames are determined, thereby effectively improving the efficiency of video title generation, so that after the user uploads the video, the corresponding video title can be generated and recommended to the user. The subsequent processing is performed by the server.

[0167] Figure 6 The system architecture diagram for the video title generation system is as follows: Figure 6 As shown, the video title generation system consists of three parts:

[0168] Part 1: Client

[0169] Clients include but are not limited to mobile applications, desktop applications, and web applications. This part is mainly responsible for local processing of videos submitted by users, such as video frame extraction, audio extraction, frame upload, audio compression and upload, and video frame content recognition (i.e., image label determination process). The processed data will be integrated into an AI recommendation task for smart titles and submitted to [Part 2: Server]. When the smart title recommendation task returns the result, the recommendation result will be presented to the user for selection.

[0170] Figure 7 The main flow chart for local client processing, such as Figure 7 As shown, after the user selects a video or uploads a video, the following processing is mainly performed:

[0171] (1) Video frame extraction: After the user selects the video file to be submitted, the system extracts several representative video frame images from the video according to predetermined rules.

[0172] (2) Audio extraction: Extract audio files from video files and perform audio cropping and transcoding according to custom rules.

[0173] (3) Perform the following processing in parallel:

[0174] Image content recognition: Use the built-in image label recognition model to analyze video frames and generate a set of labels that reflect the main content of the image.

[0175] Frame compression and upload: compress and package the video frames, and upload the compressed files to the server application.

[0176] Audio compression and upload: compress and upload the extracted audio files, and send the processing results to the server application.

[0177] (4) Creation of intelligent title recommendation tasks: Send the video format information, image content recognition results, frame file upload results, audio file upload results, etc. to the server through the interface provided by the server to create an intelligent title recommendation task.

[0178] (5) Presentation of recommended results: The client can poll and send query requests to the server to inquire whether the video title has been generated. When the server returns the recommended results of the video title, the recommended video title is displayed to the user. The user can directly select one of the titles and can further modify it.

[0179] (6) Title style switching: The client can also provide a variety of different styles of title options. Users can switch styles according to their own needs to obtain richer and more diverse title recommendations.

[0180] Part 2: Server

[0181] The main function of the server is to receive relevant data of the title recommendation task from the client, and generate the video title prompt information (Prompt) required by the large language model based on the prompt word generation algorithm. The video title prompt information, together with the video frame, audio data, etc. provided by the client, is submitted to the multimodal large language model on the AI ​​side for processing. After the multimodal large language model generates and returns the recommended video title, the server returns the video title to the client.

[0182] Smart title task management:

[0183] The server will create and manage the life cycle of the smart title task through the smart title task manager. The smart title task will pass the generated video title prompt information, video frames, audio data and other multimodal data to the artificial intelligence application to automatically generate the video title.

[0184] Smart Title Tasks:

[0185] The smart title task is constructed using the state machine design pattern. The state transition process of each task is as follows: Figure 8 The main states include the following:

[0186] Task creation: In this state, the intelligent title generation function of the artificial intelligence application is called to pass multimodal data such as video title prompt information, video frames, audio data, etc. to the multimodal large language model to generate intelligent title recommendations.

[0187] Task execution: In this state, the system polls whether the results of the recommended tasks have been generated.

[0188] Task timeout: If the result is not obtained within the specified time, the task enters the timeout state.

[0189] Task end: When the task is completed or the task times out, the task will enter the end state.

[0190] Task destruction: If the client actively cancels the task, the task will be destroyed and enter the task destruction state.

[0191] Results of the Smart Title Task:

[0192] Task success: The smart title recommendation results are successfully generated and returned.

[0193] Task failed: Smart title recommendation failed and no valid results were generated.

[0194] Task cancellation: The client actively canceled the smart title recommendation task.

[0195] Part 3: Artificial Intelligence Applications

[0196] The multimodal large language model consists of two main parts:

[0197] Model training: The AI ​​model is trained by inputting data such as video, audio, and title to provide a multimodal large language model that supports video title recommendation.

[0198] Model application: By inputting video title prompt information (Prompt), video frames, audio data, etc., the multimodal large language model can generate a video title that is highly consistent with the video content.

[0199] Fig. 9 FIG. 4 shows a structural block diagram of a video title generating device according to an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0200] The acquisition module 901 is adapted to acquire multimodal data in the video to be processed, wherein the multimodal data includes: video frames and audio data;

[0201] A determination module 902, adapted to determine a video title prompt keyword according to the multimodal data, and generate video title prompt information according to the video title prompt keyword;

[0202] The generation module 903 is adapted to input the video title prompt information and the multimodal data into a pre-trained multimodal large language model to generate a video title corresponding to the video to be processed.

[0203] In an optional implementation, the determination module is further adapted to: generate a main prompt sentence according to the video title prompt keywords and the multimodal data, and generate the video title prompt information according to the main prompt sentence and the preset prompt sentence.

[0204] In an optional implementation, the video title hint keywords include: video title style hint keywords and video title content hint keywords;

[0205] The determination module is further adapted to: determine an image tag of a video frame;

[0206] Acquire a first keyword that has a mapping relationship with the image tag, and determine the first keyword as a video title style prompt keyword;

[0207] A second keyword is extracted from the image tag, and the second keyword is determined as a video title content prompt keyword.

[0208] In an optional implementation, the video title prompt keywords include: video title content prompt keywords;

[0209] The determination module is further adapted to: extract a third keyword from the audio data and the video format information, and determine the third keyword as a video title content prompt keyword.

[0210] In an optional embodiment, the device further comprises: a screening module adapted to determine the appearance time of the image tag in the video to be processed;

[0211] Calculate the target time proportion of the image label based on the appearance time;

[0212] Filter image labels based on target time ratio;

[0213] The determination module is further adapted to: obtain a first keyword that has a mapping relationship with the filtered image tags; and extract a second keyword from the filtered image tags.

[0214] In an optional embodiment, the screening module is further adapted to: if there are multiple identical image tags, then for any one of the multiple identical image tags, the initial time ratio of the image tag is calculated according to the appearance time, and the multiple initial time ratios are added to obtain the target time ratio of the same image tag.

[0215] In an optional implementation, the device further includes: a training module adapted to obtain a sample video, video metadata corresponding to the sample video, and video comment information, and perform data annotation processing on the video metadata to obtain an annotation result;

[0216] Extract features from sample videos to obtain corresponding video features and audio features;

[0217] Generate subtitle information based on sample videos, and extract text features from subtitle information and video comment information;

[0218] The model is trained based on video features, audio features, text features and annotation results to obtain a multimodal large language model.

[0219] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0220] According to the video title generation device provided by the embodiment of the present application, by screening image tags, the accuracy of the generated video title and the efficiency of video generation can be effectively improved, and the multimodal large language model can be used in combination with the multimodal data of the video content to automatically generate the video title, so that the generated video title is highly consistent with the video content, and the generated video title can both accurately reflect the video content and attract user clicks, thereby improving the production efficiency and distribution effect of video manuscripts, improving user experience, and saving manpower and material costs; based on image tags and multimodal data, video title prompt information is generated, so that the video title prompt information and the user's manuscript content are highly matched, so that the video title generated based on the video title prompt information is highly consistent with the video content.

[0221] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program. The executable instruction or computer program can enable a processor to perform operations corresponding to the video title generation method in any of the above method embodiments.

[0222] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform operations corresponding to the video title generation method in any of the above method embodiments.

[0223] Fig.10 A schematic diagram of the structure of an embodiment of a computing device of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0224] like Fig.10 As shown, the computing device may include: a processor (processor) 1002 , a communication interface (Communications Interface) 1004 , a memory (memory) 1006 , and a communication bus 1008 .

[0225] The processor 1002, the communication interface 1004, and the memory 1006 communicate with each other via a communication bus 1008. The communication interface 1004 is used to communicate with other devices such as a client or other server network elements. The processor 1002 is used to execute the program 1010, which can specifically execute the relevant steps in the above-mentioned video title generation method embodiment for a computing device.

[0226] Specifically, the program 1010 may include program codes, which include computer operation instructions.

[0227] The processor 1002 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0228] The memory 1006 is used to store the program 1010. The memory 1006 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0229] The program 1010 can be specifically used to enable the processor 1002 to execute the video title generation method in any of the above method embodiments. The specific implementation of each step in the program 1010 can refer to the corresponding description of the corresponding steps and units in the above video title generation embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiments, which will not be repeated here.

[0230] The algorithm and display provided herein are not inherently related to any specific computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application embodiment described herein, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application embodiment.

[0231] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0232] Similarly, it should be understood that in order to streamline the present disclosure and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed embodiments of the present application require more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the embodiment of the present application.

[0233] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0234] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the embodiments of the present application and to form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0235] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0236] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The embodiments of the present application may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A method for generating a video title, comprising: Acquire multimodal data in a video to be processed, wherein the multimodal data includes: video frames and audio data; Determining a video title prompt keyword according to the multimodal data, and generating video title prompt information according to the video title prompt keyword; The video title prompt information and the multimodal data are input into a pre-trained multimodal large language model to generate a video title corresponding to the video to be processed.

2. The method according to claim 1, wherein: The step of generating video title prompt information according to the video title prompt keyword further comprises: A main prompt sentence is generated according to the video title prompt keyword and the multimodal data, and video title prompt information is generated according to the main prompt sentence and the preset prompt sentence.

3. The method according to claim 1 or 2, wherein: The video title prompt keywords include: video title style prompt keywords and video title content prompt keywords; Determining the video title prompt keywords according to the multimodal data further includes: Determining an image tag of the video frame; Acquire a first keyword that has a mapping relationship with the image tag, and determine the first keyword as a video title style prompt keyword; A second keyword is extracted from the image tag, and the second keyword is determined as a video title content prompt keyword.

4. The method according to claim 3, wherein: The video title prompt keywords include: video title content prompt keywords; Determining the video title prompt keywords according to the multimodal data further includes: A third keyword is extracted from the audio data and the video format information, and the third keyword is determined as a video title content prompt keyword.

5. The method according to claim 3 or 4, wherein: After determining the image tag of the video frame, the method further includes: Determine the appearance time of the image tag in the video to be processed; Calculate the target time ratio occupied by the image label according to the appearance time; Filter image labels according to the target time ratio; Acquiring the first keyword that has a mapping relationship with the image tag further includes: acquiring the first keyword that has a mapping relationship with the filtered image tag; The extracting the second keyword from the image tag further includes: extracting the second keyword from the filtered image tags.

6. The method according to claim 5, wherein: The step of calculating the target time ratio occupied by the image tag according to the appearance time further includes: If there are multiple identical image tags, for any one of the multiple identical image tags, the initial time proportion of the image tag is calculated according to the appearance time, and the multiple initial time proportions are added together to obtain the target time proportion of the same image tag.

7. The method according to any one of claims 1 to 6, wherein: The training method of the multimodal large language model includes: Obtaining a sample video, video metadata corresponding to the sample video, and video comment information, and performing data annotation processing on the video metadata to obtain an annotation result; Extract features from sample videos to obtain corresponding video features and audio features; generating subtitle information according to the sample video, and extracting text features from the subtitle information and video comment information; Model training is performed according to the video features, the audio features, the text features and the annotation results to obtain a multimodal large language model.

8. A video title generating device, comprising: An acquisition module, adapted to acquire multimodal data in a video to be processed, wherein the multimodal data includes: video frames and audio data; a determination module adapted to determine a video title prompt keyword according to the multimodal data, and generate video title prompt information according to the video title prompt keyword; The generation module is adapted to input the video title prompt information and the multimodal data into a pre-trained multimodal large language model to generate a video title corresponding to the video to be processed.

9. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the video title generation method according to any one of claims 1-7.

10. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the video title generation method according to any one of claims 1 to 7.

11. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the video title generation method according to any one of claims 1 to 7.

Images