Short video automatic generation method and device based on new media content creation model

Abstract

The invention discloses a short video automatic generation method and device based on a new media content creation model, and a server side method comprises the steps: receiving a short video generation request sent by a client side, wherein the short video generation request carries a video production description text and a user identifier; inputting the video production description text into a pre-fine-tuned large language model, and outputting video production scheme information; searching temporary video materials and system materials uploaded by the user from a BI system through the user identifier; under the condition that the temporary video material is not empty, inputting the video production scheme information, the temporary video material and the system material into a video generation model; or under the condition that the temporary video material is empty, inputting the video production scheme information and the system material into the video generation model; and outputting the target short video, and sending the target short video to the client. Therefore, by adopting the embodiment of the invention, the generated short video can show rich layering sense and emotion tension, so that the quality of the short video is improved.

Description

Technical Field

[0001] The present application relates to the technical field of new media content creation, and in particular to a method and device for automatically generating short videos based on a new media content creation model. Background Art

[0002] With the rapid development of self-media platforms, short videos have become an important way of information dissemination. Users use short videos to share their lives, convey knowledge, and conduct marketing promotions, and the demand for short video creation is growing. However, most users lack professional photography and video production knowledge, resulting in obvious deficiencies in the composition, color matching, and camera switching of the generated videos, which affects the visual effects and viewing experience of the videos.

[0003] In the related technology, some video production platforms provide simple templates to help users quickly generate videos. The templates provided by existing video production platforms are relatively general, lacking in personalization and artistry, and are difficult to meet the personalized needs of users in actual scenarios. At the same time, the emotional expression of videos is often relatively simple, making it difficult for the generated short videos to show rich layers and emotional tension, thereby reducing the quality of short videos. Summary of the invention

[0004] The embodiments of the present application provide a method and device for automatically generating short videos based on a new media content creation model. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a method for automatically generating short videos based on a new media content creation model, which is applied to a server, and the method includes: When receiving a short video generation request sent by a client, calling a preset new media content creation model; the short video generation request carries a video production description text and a user identifier, and the preset new media content creation model includes a pre-fine-tuned large language model and a video generation model; Input the video production description text into the pre-fine-tuned large language model, and output the video production plan information corresponding to the video production description text, the video production plan information includes video script framework information, video artistic expression method information and element supplementary description information; Through the user ID, search the BI system for system materials related to the temporary video materials uploaded by the user and the video production description text; When the temporary video material is not empty, the video production plan information, the temporary video material and the system material are input into the video generation model to generate the target short video; or, when the temporary video material is empty, the video production plan information and the system material are input into the video generation model to generate the target short video; Output the target short video corresponding to the video production description text, and send the target short video to the client.

[0006] In a second aspect, an embodiment of the present application provides a short video automatic generation device based on a new media content creation model, the device comprising: A model calling module is used to call a preset new media content creation model upon receiving a short video generation request sent by a client; the short video generation request carries a video production description text and a user identifier, and the preset new media content creation model includes a pre-fine-tuned large language model and a video generation model; A text processing module is used to input the video production description text into a pre-fine-tuned large language model, and output video production plan information corresponding to the video production description text, where the video production plan information includes video script framework information, video artistic expression method information, and element supplementary description information; The material calling module is used to search the temporary video materials uploaded by the user and the system materials related to the video production description text from the BI system through the user ID; A short video generation module, for inputting the video production scheme information, the temporary video material and the system material into the video generation model to generate a target short video when the temporary video material is not empty; or, for inputting the video production scheme information and the system material into the video generation model to generate a target short video when the temporary video material is empty; The video sending module is used to output the target short video corresponding to the video production description text and send the target short video to the client.

[0007] The technical solution provided by the embodiments of the present application may have the following beneficial effects: In an embodiment of the present application, on the one hand, in the short video generation process, a pre-fine-tuned large language model and a video generation model are used. The large language model can better understand the semantics of the video production description text through fine-tuning, thereby generating a more accurate and professional video production plan. The video generation model is based on the video production plan and related materials, and uses advanced generation technology to generate high-quality video content, so that the generated short video shows a rich sense of hierarchy and emotional tension, so that the generated short video is more attractive visually and auditorily, thereby improving the quality of the short video. On the other hand, the video generation model can make adaptive selections when the temporary video material is empty or not empty, and then can adaptively generate a target short video that meets the user's actual scene, meeting the personalized needs of the user's actual scene, thereby improving the flexibility of generating short videos.

[0008] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0010] Figure 1 It is a method flow diagram of a method for automatically generating short videos based on a new media content creation model provided in an embodiment of the present application; Figure 2 It is a user interface diagram displayed by a client provided in an embodiment of the present application; Figure 3 It is a schematic diagram of a model architecture of a video generation model provided in an embodiment of the present application; Figure 4 This is a schematic diagram of interaction between a server and a client provided in an embodiment of the present application; Figure 5 It is a flowchart of a model generation method of a pre-fine-tuned large language model provided in an embodiment of the present application; Figure 6 It is a structural schematic diagram of a short video automatic generation device based on a new media content creation model provided in an embodiment of the present application; Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.

[0012] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0013] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0014] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0015] At present, the creation of video content mainly relies on some video production platforms to provide simple templates to help users quickly generate videos.

[0016] The inventors realized that the templates provided by existing video production platforms are relatively general, lacking in personalization and artistry, and are difficult to meet the personalized needs of users in actual scenarios. At the same time, the emotional expression of videos is often relatively simple, making it difficult for the generated short videos to show rich layers and emotional tension, thereby reducing the quality of the short videos.

[0017] In order to solve the above problems, the present application provides a method and device for automatically generating short videos based on a new media content creation model to solve the problems existing in the above-mentioned related technical problems. In the embodiment of the present application, on the one hand, in the short video generation process, a pre-fine-tuned large language model and a video generation model are used. The large language model can better understand the semantics of the video production description text through fine-tuning, thereby generating a more accurate and professional video production plan. The video generation model is based on the video production plan and related materials, and uses advanced generation technology to generate high-quality video content, so that the generated short video shows a rich sense of hierarchy and emotional tension, so that the generated short video is more attractive visually and auditorily, thereby improving the quality of the short video. On the other hand, the video generation model can make adaptive selections when the temporary video material is empty or not empty, and then can adaptively generate a target short video that meets the user's actual scene, meet the personalized needs of the user's actual scene, thereby improving the flexibility of generating short videos, and the following exemplary embodiments are used for detailed description.

[0018] The following will be combined with the attached Figure 1 -Attached Figure 5 , the method for automatically generating short videos based on the new media content creation model provided by the embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a short video automatic generation device based on the new media content creation model based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0019] See also Figure 1 , which is a flow chart of a method for automatically generating short videos based on a new media content creation model, and is applied to a server. Figure 1 As shown, the method of the embodiment of the present application may include the following steps: S101, upon receiving a short video generation request sent by a client, calling a preset new media content creation model; the short video generation request carries a video production description text and a user identifier, and the preset new media content creation model includes a pre-fine-tuned large language model and a video generation model; Among them, the client is a device or software used by the user to send requests and receive generated videos. For example, mobile applications, web interfaces, etc. A short video generation request is a request initiated by a user to generate a short video. The request contains necessary information, such as video production description text and user identification. The preset new media content creation model is a pre-configured model system for generating short videos, which includes two main parts: a pre-fine-tuned large language model and a video generation model. User identification is information used to identify the user, such as a user ID or account number. The pre-fine-tuned large language model is a fine-tuned large language model used to understand the video production description text and generate video production plan information. The video generation model is an algorithmic model used to generate the final video content based on the video production plan information and materials.

[0020] In some embodiments of the present application, first, the user uploads a temporary video material in the BI system, then the user enters a video production description text in the client (such as a mobile application) and clicks Submit. The server receives the short video generation request sent by the client, parses the request content, extracts the video production description text and user ID, and finally the server calls a preset new media content creation model, which includes a pre-fine-tuned large language model and a video generation model.

[0021] For example Figure 2 As shown in the figure, the user wants to send a short video generation request through the client to generate a short video with the theme of "Li Shangyin's "Night Rain Sent to the North"". The user provides a detailed video production description text and uploads some temporary video materials (such as rain scenes, candlelight, etc.) in the BI system. The video production description text is, for example: "Please make a 30-second video based on Li Shangyin's poem "Night Rain Sent to the North".

[0022] S102, inputting the video production description text into a pre-fine-tuned large language model, and outputting video production plan information corresponding to the video production description text, where the video production plan information includes video script framework information, video artistic expression technique information, and element supplementary description information; Among them, the video production plan information is the detailed video production plan and guidance information generated by the pre-fine-tuned large language model based on the video production description text. The video script framework information is the specific structure and content arrangement of the video, describing the specific scenes and content of each part of the video such as the beginning, middle and end. The artistic expression technique information of the video is used to enhance the various technologies and methods of the artistic effect and emotional expression of the video. The element supplementary description information is a detailed description and design requirements of specific elements in the video, which is used to further refine the visual and auditory effects of the video.

[0023] In some embodiments of the present application, after receiving the video production description text, the server inputs it into a pre-fine-tuned large language model, and the pre-fine-tuned large language model generates detailed video production plan information based on the input video production description text.

[0024] For example, the video production description text is: "Please make a 30-second video based on Li Shangyin's poem "Night Rain Sent to the North". After processing by the pre-fine-tuned large language model, the video production plan information can be viewed in the server's log information, as shown below: The log information in JSON format is as follows: script framework:{ opening: The camera opens with a close-up of raindrops sliding down bamboo leaves in the dark night, showing the panoramic view of Bashan Mountain in autumn night, with rain curtains covering the mountains and a silhouette of a single lamp swaying in front of the window. The subtitles gradually appear, "You ask when I will return, but there is no date."

[0025] middle: Dynamic transition: Raindrops fall into the pond, ripples spread, and then turn into a close-up of "the water in the autumn pond rises". Contrast between reality and illusion: The camera switches to the poet's profile staring at the candlelight, the picture gradually blurs, and an imaginary picture emerges - under the west window, the candlelight is warm, and the poet and his relatives and friends are talking by candlelight. " ending: The camera pulls back to reality, the sound of rain fades, the candlelight goes out, the morning light is faintly visible outside the window, and the subtitles freeze "Let's talk about the night rain in Bashan Mountain".} artistic techniques: audio:"The background sound is a zither and flute ensemble, with the sound of rain gradually weakening, reinforcing the sense of loneliness and longing. The reading adopts the rhythm of "meaning division" (such as "You ask / when to return / there is no date"), with a slow tone and a slightly rising last sentence, reflecting the tension of time and space.

[0026] Visual symbols: Rain: symbolizes continuous sadness, and conveys emotions through details such as close-ups of raindrops and water ripples. Candlelight: The lonely lamp in reality and the imagined "cut candle" form a cold and warm contrast, metaphorically implying the conflict between hope and reality. Letter: Close-up of the word "return date" written by a brush, with ink smudged, implying an unfulfilled promise.} element supplements: subtitle_design: "Using running script font, the poem lines appear one by one, matching the rhythm of the picture, and the last line stays for 3 seconds." color_contrast: "The real part is mainly blue-gray, while the imaginary scene turns to warm yellow to highlight the emotional contrast.} Specifically, the video script framework information included in the video production plan information includes the following: Beginning (0-5 seconds): The camera pulls away from a close-up of raindrops sliding down bamboo leaves in the dark night, showing the panoramic view of the autumn night in Bashan, the rain curtain covering the mountains, the silhouette of a lonely lamp swaying in front of the window, and the subtitles gradually show "You ask about the return date, but there is no date yet." Middle section (6-20 seconds): Dynamic transition: Raindrops fall into the pond, ripples spread, and superimposed to a close-up of "the water in the autumn pond rises." Contrast between reality and illusion: The camera switches to the poet's profile staring at the candlelight, the picture gradually becomes blurred, and an imaginary picture emerges - under the west window, the candlelight is warm, and the poet and his relatives and friends cut the candle and talk together. Ending (21-30 seconds): The camera pulls back to reality, the sound of rain gradually weakens, the candlelight goes out, the morning light outside the window is slightly exposed, and the subtitles freeze "Let's talk about the night rain in Bashan."

[0027] The video production plan information includes the artistic expression techniques of the video, including sound effects and music: the background sound uses the ensemble of zither and flute, and the sound of rain gradually weakens, which strengthens the sense of loneliness and longing. The reading adopts the rhythm of "meaning division" (such as "You ask / return date / no date"), with a slow tone and a slightly rising end, reflecting the tension of time and space. Visual symbols: Rain, symbolizing continuous melancholy, conveys emotions through details such as close-ups of raindrops and water ripples. Candlelight, the lonely lamp in reality and the imagined "cut candle" form a contrast between cold and warm, metaphorically implying the conflict between hope and reality. Letters, close-ups of the word "return date" written with a brush, and the ink stains are blurred, implying an unfulfilled promise.

[0028] The supplementary information of the elements included in the video production plan includes: Subtitle design: Use running script font, the poem lines appear one by one, match the rhythm of the picture, and stay for 3 seconds at the end. Tone contrast: The reality part is mainly blue-gray, and the imaginary scene turns to warm yellow to highlight the emotional contrast.

[0029] In some embodiments of the present application, the specific process of generating a pre-fine-tuned large language model includes: obtaining video creation requirements for short video generation tasks and a preset video production solution knowledge base, the video creation requirements are used to characterize the basic requirements for creating short videos, and the preset video production solution knowledge base is constructed based on standard data sets related to short video creation, and the standard data sets related to short video creation include video scripts, scene description data, and emotional expression data; using the pre-trained preset large language model as the basic model; modifying the basic model according to the video creation requirements of the short video generation task to determine the target parameters suitable for the video creation requirements; fine-tuning and optimizing the basic model through the sample data and target parameters in the video production solution knowledge base to obtain the pre-fine-tuned large language model.

[0030] In the embodiment of the present application, by obtaining the video creation requirements and the preset video production plan knowledge base of the short video generation task, and making targeted modifications and fine-tuning to the pre-trained large language model based on these data, the performance of the model in the short video creation task can be significantly improved. This fine-tuning process enables the model to better understand the specific requirements of video creation, generate video scripts and production plans that better meet user needs, and thus improve the quality and efficiency of short video creation.

[0031] Specifically, the basic model is fine-tuned and optimized through the sample data and target parameters in the video production solution knowledge base to obtain the pre-fine-tuned large language model. The specific process includes: dividing the sample data in the video production solution knowledge base according to a preset ratio to obtain a first training set and a second training set; decomposing the target parameters to obtain a first parameter decomposition matrix and a second parameter decomposition matrix, the first parameter decomposition matrix includes parameters related to the columns of the original model parameter matrix in the basic model, and the second parameter decomposition matrix includes parameters related to the rows of the original model parameter matrix in the basic model; according to the training data in the first training set, the first parameter decomposition matrix and the second parameter decomposition matrix, the back propagation result of the basic model is calculated; according to the back propagation result and the preset target loss function, the model loss value of the basic model is calculated; when the model loss value reaches the minimum, the fine-tuned basic model is obtained; using the second training set, the fine-tuned basic model is optimized twice to obtain the pre-fine-tuned large language model.

[0032] Specifically, the calculation formula of the back propagation result is:

[0033] in, For the The back propagation result of training data is is the original parameter matrix of the basic model, is the bias, is the first parameter decomposition matrix, is the second parameter decomposition matrix, satisfying , For the training data; The preset target loss function is:

[0034] in, is the model loss value of the basic model, is the number of samples of training data, is the number of categories, It is training data in the category The true label on is 0 or 1. It is training data in the category The predicted probability of is the weight of the regularization term, is the back propagation result, It is the model parameter after the basic model integrates the back propagation results Gradient The square of the norm.

[0035] Specifically, the specific process of using the second training set to perform secondary optimization on the fine-tuned basic model includes: inputting each training set in the second training set into the fine-tuned basic model, and outputting the video production plan information corresponding to each training set; analyzing the performance quantification value of the fine-tuned basic model according to the video production plan information; determining the evaluation value according to the performance quantification value and in combination with a preset evaluation function; when the evaluation value does not reach the preset expected value, updating the parameters of the fine-tuned basic model to maximize the expectation of the evaluation value determined by the preset evaluation function.

[0036] Among them, the update can use the policy gradient method in reinforcement learning, and the evaluation function is: , is the evaluation value, is a constant used to adjust the range of the evaluation value. The performance quantification value can be determined by analyzing the difference between the video production plan information corresponding to each training set and the label of each training set.

[0037] S103, searching the BI system for system materials related to the temporary video materials uploaded by the user and the video production description text according to the user identifier; Among them, the BI system is a Business Intelligence system, which is used to store and manage video materials uploaded by users and system materials. For example, it is an enterprise-level multimedia material management system, where users can upload and manage their own video materials. Temporary video materials are temporary video materials uploaded by users, which are usually used for specific video generation tasks and can be deleted after the task is completed. System materials are preset video material libraries, which contain various common video clips, background music, fonts, etc., and are used to assist video generation.

[0038] In some embodiments of the present application, after obtaining the video production plan information output by the pre-fine-tuned large language model, the server searches the BI system for temporary video materials uploaded by the user through the user ID (user ID user123), and then obtains the semantics of the video production description text, and searches the preset material library of the BI system for system materials whose semantic similarity with the video production description text is greater than a threshold, thereby obtaining system materials related to the video production description text.

[0039] S104, when the temporary video material is not empty, input the video production plan information, the temporary video material and the system material into the video generation model to generate the target short video; or, when the temporary video material is empty, input the video production plan information and the system material into the video generation model to generate the target short video; The video script framework information includes a description of the beginning scene of the video, a description of the middle scene of the video, and a description of the ending scene of the video.

[0040] For example Figure 3 As shown, the video generation model includes a semantic vector extraction module, a material feature extraction module, a material matching module, a video clip generation module, a video splicing module and a video optimization module.

[0041] Among them, if the temporary video material is not empty, it means that the current user has shot relevant content for generating a short video, and the shot content may be shot by a professional photographer or an ordinary user.

[0042] In some embodiments of the present application, the specific process of generating a target short video includes: a semantic vector extraction module uses natural language processing (NLP) technology to parse the scene description at the beginning of the video, the scene description in the middle of the video, and the scene description at the end of the video to extract semantic vectors for key scene descriptions; a material feature extraction module classifies and annotates image frames of temporary video materials and system materials when the temporary video material is not empty to extract a first material feature of the temporary video material and a second material feature of the system material; the first material feature and the second material feature of the system material are composed of visual features and audio features; a material matching module performs material matching on the first material feature and the second material feature according to the semantic vector to obtain a first matching result and a second matching result; The first matching result has a first target feature corresponding to part or all of the semantic vectors, and the second matching result has a second target feature corresponding to all of the semantic vectors; the video clip generation module generates multiple video clips suitable for the scene description at the beginning of the video, the scene description in the middle of the video, and the scene description at the end of the video based on the first matching result and the second matching result; the video splicing module splices multiple video clips based on the artistic expression technique information to obtain an initial short video; the artistic expression technique information includes sound effect and music information, visual symbol information, and editing technique information; the video optimization module designs subtitles and performs color tone conversion on the initial short video based on the element supplementary description information to obtain a target short video; the element supplementary description information includes subtitle design and color tone contrast information.

[0043] It should be noted that the first matching result has a first target feature corresponding to part of the semantic vectors, indicating that the current user is an ordinary user, that is, some content in the temporary video material meets the requirements, and some content does not meet the requirements. The first matching result has a first target feature corresponding to all the semantic vectors, indicating that the current user is a professional photographer, that is, all the content in the temporary video material meets the requirements. The reason why the second matching result has a second target feature corresponding to all the semantic vectors is that the system materials are all standardized, so all of them meet the requirements.

[0044] The semantic vectors include the semantic vectors of the scene at the beginning of the video, the semantic vectors of the scene in the middle of the video, and the semantic vectors of the scene at the end of the video.

[0045] In some embodiments of the present application, the specific process of generating multiple video clips suitable for describing the scene at the beginning of the video, describing the scene in the middle of the video, and describing the scene at the end of the video based on the first matching result and the second matching result includes: when the first matching result has a first target feature corresponding to all the semantic vectors, obtaining material content corresponding to the first target feature corresponding to all the semantic vectors from the temporary video material; or, when the first matching result has a first target feature corresponding to part of the semantic vectors, obtaining first material content to be analyzed corresponding to the first target feature corresponding to part of the semantic vectors from the temporary video material; determining the semantic vector to be analyzed that is not matched with the first material feature; and determining, from the second target features corresponding to all the semantic vectors that exist in the second matching result, the first material content to be analyzed that is corresponding to the first target feature corresponding to part of the semantic vectors. corresponding to the second target feature; obtaining the second material content to be analyzed corresponding to the second target feature corresponding to the semantic vector to be analyzed from the system material; constructing the material content corresponding to all the semantic vectors according to the first material content to be analyzed and the second material content to be analyzed; classifying the above material content to obtain the material content corresponding to the semantic vector of the scene at the beginning of the video, the material content corresponding to the semantic vector of the scene in the middle of the video and the material content corresponding to the semantic vector of the scene at the end of the video; using the material content corresponding to the semantic vector of the scene at the beginning of the video as a video clip suitable for describing the scene at the beginning of the video; using the material content corresponding to the semantic vector of the scene in the middle of the video as a video clip suitable for describing the scene in the middle of the video; using the material content corresponding to the semantic vector of the scene at the end of the video as a video clip suitable for describing the scene at the end of the video.

[0046] In an embodiment of the present application, when the temporary video material is not empty, it is necessary to generate a short video based on the style of the temporary video material uploaded by the user. Therefore, when the first matching result has a first target feature corresponding to all semantic vectors, all content in the temporary video material meets the requirements, and the temporary video material needs to be used to make a short video. When the first matching result has a first target feature corresponding to part of the semantic vectors, part of the content in the temporary video material meets the requirements. At this time, it is necessary to obtain the system material corresponding to the temporary material that does not meet the requirements from the system material, and then transform the system material corresponding to the temporary material that does not meet the requirements based on the style of the temporary material content that meets the requirements, so that the system material that meets the requirements becomes a material suitable for the user's style.

[0047] Specifically, the specific process of constructing material content corresponding to all semantic vectors based on the first material content to be analyzed and the second material content to be analyzed includes: analyzing the image features of the first material content to be analyzed; using the image features as algorithm parameters of a preset generation algorithm, and adjusting the image style of the second material content to be analyzed through the preset generation algorithm to obtain optimized second material content to be analyzed; the generation algorithm is a diffusion model; and the first material content to be analyzed and the optimized second material content to be analyzed are merged into the material content corresponding to all semantic vectors.

[0048] In an embodiment of the present application, when the temporary video material is not empty, the system material is modified based on the style of the temporary video material uploaded by the user to make it suitable for the current user, and a target short video that meets the user's actual scenario is generated, thereby meeting the personalized needs of the user's actual scenario.

[0049] In other embodiments of the present application, the specific process of generating a target short video includes: when the temporary video material is empty, classifying and labeling the image frames of the system material to extract the second material feature of the system material; performing material matching on the second material feature according to the semantic vector to obtain multiple video clips suitable for the scene description at the beginning of the video, the scene description in the middle of the video, and the scene description at the end of the video; based on the artistic expression technique information, performing video splicing on the multiple video clips to obtain an initial short video; based on the element supplementary description information, performing subtitle design and color tone conversion on the initial short video to obtain the target short video.

[0050] It should be noted that when the temporary video material is empty, it means that the user has not uploaded any material and only generates short videos based on text. At this time, there is no need to rely on the user's video style, and there is no need to make any modifications to the system material. It can be directly based on the image frames of the system material to match multiple video clips suitable for the scene description at the beginning of the video, the scene description in the middle of the video, and the scene description at the end of the video.

[0051] In an embodiment of the present application, based on the information of artistic expression techniques, multiple video clips are spliced ​​to obtain the specific process of the initial short video: each video clip is processed based on the sound effect and music information, visual symbol information, and editing skill information, and the processed video clips are spliced ​​to obtain the initial short video. For example, in the video generation based on "Night Rain Sent to the North", the sound effect and music information: the background sound uses the ensemble of guzheng and flute, and the sound of rain gradually weakens. Visual symbol information: Rain symbolizes continuous melancholy, and candlelight forms a contrast between cold and warm. Editing skill information: Use "superimposition" to connect real and imaginary scenes, alternating between fast and slow shots.

[0052] In the embodiment of the present application, based on the element supplementary description information, the initial short video is subtitled and the color tone is converted. In the process of obtaining the target short video, the element supplementary description information includes subtitle design and color tone contrast information. For example, in the video generation based on "Night Rain Sent to the North", the subtitle design: using running script font, the poem emerges sentence by sentence, and the last sentence stays for 3 seconds. Color tone contrast: the reality part is mainly blue-gray, and the imaginary scene turns to warm yellow.

[0053] S105, output the target short video corresponding to the video production description text, and send the target short video to the client.

[0054] In the embodiment of the present application, after obtaining the target short video, the generated video can be sent back to the client through the backend service, and the user can view and download the generated short video on the front-end interface. The front-end page will display a video player, and the user can play the generated short video. At this time, the video content is consistent with the theme of "Night Rain Sends North", with rich layers and emotional tension.

[0055] For example Figure 4 As shown, the server 110 can be a server, which can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, such as server devices for running new media content creation models and BI systems. When it is necessary to automatically generate a short video, the server 110 calls a preset new media content creation model upon receiving a short video generation request sent by the client 120; the short video generation request carries a video production description text and a user identifier, and the preset new media content creation model includes a pre-adjusted large language model and a video generation model; the video production description text is input into the pre-adjusted large language model, and video production plan information corresponding to the video production description text is output, the video production plan information includes video script framework information, video artistic expression technique information and element supplementary description information; through the user identifier, the temporary video material uploaded by the user and the system material related to the video production description text are searched from the BI system; when the temporary video material is not empty, the video production plan information, the temporary video material and the system material are input into the video generation model to generate a target short video; or, when the temporary video material is empty, the video production plan information and the system material are input into the video generation model to generate a target short video; the target short video corresponding to the video production description text is output, and the target short video is sent to the client 120.

[0056] It should be noted that the client 120 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The server 110 and the client 120 may be connected via Bluetooth, USB (Universal Serial Bus) or other communication connection methods, and the present invention does not limit this.

[0057] In an embodiment of the present application, on the one hand, in the short video generation process, a pre-fine-tuned large language model and a video generation model are used. The large language model can better understand the semantics of the video production description text through fine-tuning, thereby generating a more accurate and professional video production plan. The video generation model is based on the video production plan and related materials, and uses advanced generation technology to generate high-quality video content, so that the generated short video shows a rich sense of hierarchy and emotional tension, so that the generated short video is more attractive visually and auditorily, thereby improving the quality of the short video. On the other hand, the video generation model can make adaptive selections when the temporary video material is empty or not empty, and then can adaptively generate a target short video that meets the user's actual scene, meeting the personalized needs of the user's actual scene, thereby improving the flexibility of generating short videos.

[0058] See also Figure 5 , is a flow chart of a method for generating a model of a pre-fine-tuned large language model provided in an embodiment of the present application. Figure 5 As shown, the method of the embodiment of the present application may include the following steps: S201, obtaining video creation requirements and a preset video production solution knowledge base for a short video generation task, where the video creation requirements are used to characterize the basic requirements for creating a short video, and the preset video production solution knowledge base is constructed based on a standard data set related to short video creation, and the standard data set related to short video creation includes video scripts, scene description data, and emotion expression data; S202, using a pre-trained preset large language model as a basic model; S203, modifying the basic model according to the video creation requirement of the short video generation task to determine target parameters suitable for the video creation requirement; S204, fine-tuning and optimizing the basic model through sample data and target parameters in the video production solution knowledge base to obtain a pre-fine-tuned large language model.

[0059] In an embodiment of the present application, on the one hand, in the short video generation process, a pre-fine-tuned large language model and a video generation model are used. The large language model can better understand the semantics of the video production description text through fine-tuning, thereby generating a more accurate and professional video production plan. The video generation model is based on the video production plan and related materials, and uses advanced generation technology to generate high-quality video content, so that the generated short video shows a rich sense of hierarchy and emotional tension, so that the generated short video is more attractive visually and auditorily, thereby improving the quality of the short video. On the other hand, the video generation model can make adaptive selections when the temporary video material is empty or not empty, and then can adaptively generate a target short video that meets the user's actual scene, meeting the personalized needs of the user's actual scene, thereby improving the flexibility of generating short videos.

[0060] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0061] See also Figure 6 , which shows a schematic diagram of the structure of a short video automatic generation device based on a new media content creation model provided by an exemplary embodiment of the present application. The short video automatic generation device based on a new media content creation model can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device 1 includes a model calling module 10, a text processing module 20, a material calling module 30, a short video generation module 40, and a video sending module 50.

[0062] The model calling module 10 is used to call the preset new media content creation model when receiving a short video generation request sent by the client; the short video generation request carries a video production description text and a user identifier, and the preset new media content creation model includes a pre-fine-tuned large language model and a video generation model; The text processing module 20 is used to input the video production description text into the pre-fine-tuned large language model, and output the video production plan information corresponding to the video production description text, the video production plan information includes video script framework information, video artistic expression method information and element supplementary description information; The material calling module 30 is used to search the temporary video material uploaded by the user and the system material related to the video production description text from the BI system according to the user identification; The short video generation module 40 is used to input the video production scheme information, the temporary video material and the system material into the video generation model to generate a target short video when the temporary video material is not empty; or, when the temporary video material is empty, input the video production scheme information and the system material into the video generation model to generate a target short video; The video sending module 50 is used to output the target short video corresponding to the video production description text and send the target short video to the client.

[0063] It should be noted that the automatic short video generation device based on the new media content creation model provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the automatic short video generation method based on the new media content creation model. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the automatic short video generation device based on the new media content creation model provided in the above embodiment and the automatic short video generation method based on the new media content creation model belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0064] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0065] In an embodiment of the present application, on the one hand, in the short video generation process, a pre-fine-tuned large language model and a video generation model are used. The large language model can better understand the semantics of the video production description text through fine-tuning, thereby generating a more accurate and professional video production plan. The video generation model is based on the video production plan and related materials, and uses advanced generation technology to generate high-quality video content, so that the generated short video shows a rich sense of hierarchy and emotional tension, so that the generated short video is more attractive visually and auditorily, thereby improving the quality of the short video. On the other hand, the video generation model can make adaptive selections when the temporary video material is empty or not empty, and then can adaptively generate a target short video that meets the user's actual scene, meeting the personalized needs of the user's actual scene, thereby improving the flexibility of generating short videos.

[0066] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the method for automatically generating short videos based on the new media content creation model provided in the above-mentioned method embodiments.

[0067] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the automatic short video generation method based on the new media content creation model of each of the above-mentioned method embodiments.

[0068] See also Figure 7 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0069] The communication bus 1002 is used to realize the connection and communication between these components.

[0070] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0071] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0072] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0073] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 7 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a short video automatic generation application based on a new media content creation model.

[0074] exist Figure 7 In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call the short video automatic generation application based on the new media content creation model stored in the memory 1005, and specifically perform the following operations: When receiving a short video generation request sent by a client, calling a preset new media content creation model; the short video generation request carries a video production description text and a user identifier, and the preset new media content creation model includes a pre-fine-tuned large language model and a video generation model; Input the video production description text into the pre-fine-tuned large language model, and output the video production plan information corresponding to the video production description text, the video production plan information includes video script framework information, video artistic expression method information and element supplementary description information; Through the user ID, search the BI system for system materials related to the temporary video materials uploaded by the user and the video production description text; When the temporary video material is not empty, the video production plan information, the temporary video material and the system material are input into the video generation model to generate the target short video; or, when the temporary video material is empty, the video production plan information and the system material are input into the video generation model to generate the target short video; Output the target short video corresponding to the video production description text, and send the target short video to the client.

[0075] In one embodiment, when the processor 1001 generates a target short video, the processor 1001 specifically performs the following operations: The semantic vector extraction module uses natural language processing (NLP) technology to parse the scene descriptions at the beginning, middle and end of the video to extract semantic vectors for key scene descriptions. The material feature extraction module classifies and annotates the image frames of the temporary video material and the system material when the temporary video material is not empty, so as to extract the first material feature of the temporary video material and the second material feature of the system material; the first material feature and the second material feature of the system material are composed of visual features and audio features; The material matching module performs material matching on the first material feature and the second material feature respectively according to the semantic vector to obtain a first matching result and a second matching result; the first matching result has a first target feature corresponding to part or all of the semantic vectors, and the second matching result has a second target feature corresponding to all of the semantic vectors; The video segment generation module generates a plurality of video segments suitable for describing the scene at the beginning of the video, describing the scene in the middle of the video, and describing the scene at the end of the video based on the first matching result and the second matching result; The video splicing module splices multiple video clips based on the artistic expression information to obtain an initial short video; the artistic expression information includes sound effect and music information, visual symbol information and editing technique information; The video optimization module performs subtitle design and color tone conversion on the initial short video based on the element supplementary description information to obtain the target short video; the element supplementary description information includes subtitle design and color tone contrast information.

[0076] In one embodiment, when the processor 1001 generates a plurality of video segments applicable to the description of the beginning scene of the video, the description of the middle scene of the video, and the description of the ending scene of the video based on the first matching result and the second matching result, the processor 1001 specifically performs the following operations: When the first matching result has a first target feature corresponding to all the semantic vectors, obtaining material content corresponding to the first target feature corresponding to all the semantic vectors from the temporary video material; Alternatively, when the first matching result has a first target feature corresponding to a part of the semantic vector, obtain the first material content to be analyzed corresponding to the first target feature corresponding to the part of the semantic vector from the temporary video material; determine the semantic vector to be analyzed that is not matched to the first material feature; determine the second target feature corresponding to the semantic vector to be analyzed from the second target features corresponding to all the semantic vectors that exist in the second matching result; obtain the second material content to be analyzed corresponding to the second target feature corresponding to the semantic vector to be analyzed from the system material; and construct the material content corresponding to all the semantic vectors based on the first material content to be analyzed and the second material content to be analyzed; Classify the above material contents to obtain the material contents corresponding to the semantic vector of the scene at the beginning of the video, the material contents corresponding to the semantic vector of the scene in the middle of the video, and the material contents corresponding to the semantic vector of the scene at the end of the video; The material content corresponding to the semantic vector of the scene at the beginning of the video is used as a video clip suitable for describing the scene at the beginning of the video; the material content corresponding to the semantic vector of the scene in the middle of the video is used as a video clip suitable for describing the scene in the middle of the video; the material content corresponding to the semantic vector of the scene at the end of the video is used as a video clip suitable for describing the scene at the end of the video.

[0077] In one embodiment, when the processor 1001 constructs the material content corresponding to all the semantic vectors according to the first material content to be analyzed and the second material content to be analyzed, the processor 1001 specifically performs the following operations: Analyzing image features of the first material to be analyzed; The image features are used as algorithm parameters of a preset generation algorithm, and the image style of the second material content to be analyzed is adjusted by the preset generation algorithm to obtain optimized second material content to be analyzed; the generation algorithm is a diffusion model; The first material content to be analyzed and the optimized second material content to be analyzed are merged into material content corresponding to all semantic vectors.

[0078] In one embodiment, when the processor 1001 generates a target short video, the processor 1001 specifically performs the following operations: When the temporary video material is empty, classifying and labeling the image frames of the system material to extract the second material feature of the system material; Perform material matching on the second material feature according to the semantic vector to obtain multiple video clips suitable for describing the scene at the beginning of the video, describing the scene in the middle of the video, and describing the scene at the end of the video; Based on the information of artistic expression techniques, multiple video clips are spliced ​​to obtain an initial short video; Based on the element supplementary description information, the initial short video is subtitled and the color tone is converted to obtain the target short video.

[0079] In one embodiment, when the processor 1001 generates a pre-fine-tuned large language model, the processor 1001 specifically performs the following operations: Obtain the video creation requirements and preset video production solution knowledge base of the short video generation task. The video creation requirements are used to characterize the basic requirements for creating short videos. The preset video production solution knowledge base is built based on standard datasets related to short video creation. The standard datasets related to short video creation include video scripts, scene description data, and emotion expression data. Use the pre-trained preset large language model as the basic model; According to the video creation requirements of the short video generation task, the basic model is modified to determine the target parameters suitable for the video creation requirements; The basic model is fine-tuned and optimized through sample data and target parameters in the video production solution knowledge base to obtain a pre-fine-tuned large language model.

[0080] In one embodiment, when the processor 1001 fine-tunes and optimizes the basic model using the sample data and target parameters in the video production solution knowledge base to obtain the pre-fine-tuned large language model, the processor 1001 specifically performs the following operations: Dividing the sample data in the video production solution knowledge base according to a preset ratio to obtain a first training set and a second training set; Decomposing the target parameters to obtain a first parameter decomposition matrix and a second parameter decomposition matrix, wherein the first parameter decomposition matrix includes parameters related to columns of the original model parameter matrix in the base model, and the second parameter decomposition matrix includes parameters related to rows of the original model parameter matrix in the base model; Calculate the back propagation result of the basic model according to the training data in the first training set, the first parameter decomposition matrix and the second parameter decomposition matrix; Calculate the model loss value of the basic model based on the back propagation result and the preset target loss function; When the model loss value reaches the minimum, the fine-tuned basic model is obtained; The fine-tuned base model is optimized twice using the second training set to obtain the pre-fine-tuned large language model.

[0081] In one embodiment, when the processor 1001 performs secondary optimization on the fine-tuned basic model using the second training set, the processor 1001 specifically performs the following operations: Input each training set in the second training set into the fine-tuned basic model, and output the video production plan information corresponding to each training set; According to the video production plan information, analyze the performance quantitative value of the fine-tuned basic model; Determine the evaluation value according to the performance quantification value and the preset evaluation function; When the evaluation value does not reach the preset expected value, the parameters of the fine-tuned basic model are updated to maximize the expectation of the evaluation value determined by the preset evaluation function.

[0082] In an embodiment of the present application, on the one hand, in the short video generation process, a pre-fine-tuned large language model and a video generation model are used. The large language model can better understand the semantics of the video production description text through fine-tuning, thereby generating a more accurate and professional video production plan. The video generation model is based on the video production plan and related materials, and uses advanced generation technology to generate high-quality video content, so that the generated short video shows a rich sense of hierarchy and emotional tension, so that the generated short video is more attractive visually and auditorily, thereby improving the quality of the short video. On the other hand, the video generation model can make adaptive selections when the temporary video material is empty or not empty, and then can adaptively generate a target short video that meets the user's actual scene, meeting the personalized needs of the user's actual scene, thereby improving the flexibility of generating short videos.

[0083] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program for automatically generating short videos based on the new media content creation model can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program for automatically generating short videos based on the new media content creation model can be a disk, an optical disk, a read-only storage memory, or a random access storage memory, etc.

[0084] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A short video automatic generation method based on a new media content creation model, characterized in that: Applied to the server, the method includes: Upon receiving a short video generation request sent by a client, calling a preset new media content creation model; the short video generation request carries a video production description text and a user identifier, and the preset new media content creation model includes a pre-fine-tuned large language model and a video generation model; Input the video production description text into a pre-fine-tuned large language model, and output video production plan information corresponding to the video production description text, wherein the video production plan information includes video script framework information, video artistic expression technique information, and element supplementary description information; By using the user ID, searching the BI system for system materials related to the temporary video materials uploaded by the user and the video production description text; When the temporary video material is not empty, the video production scheme information, the temporary video material and the system material are input into the video generation model to generate a target short video; or, when the temporary video material is empty, the video production scheme information and the system material are input into the video generation model to generate a target short video; Output the target short video corresponding to the video production description text, and send the target short video to the client.

2. The method according to claim 1, characterized in that The video generation model includes a semantic vector extraction module, a material feature extraction module, a material matching module, a video segment generation module, a video splicing module and a video optimization module, and the video script framework information includes a description of the scene at the beginning of the video, a description of the scene in the middle of the video and a description of the scene at the end of the video; The generating of the target short video comprises: The semantic vector extraction module uses natural language processing (NLP) technology to parse the scene description at the beginning of the video, the scene description in the middle of the video, and the scene description at the end of the video to extract semantic vectors for key scene descriptions; The material feature extraction module classifies and annotates the image frames of the temporary video material and the system material when the temporary video material is not empty, so as to extract a first material feature of the temporary video material and a second material feature of the system material; the first material feature and the second material feature of the system material are composed of visual features and audio features; The material matching module performs material matching on the first material feature and the second material feature respectively according to the semantic vector to obtain a first matching result and a second matching result; the first matching result has a first target feature corresponding to part or all of the semantic vectors, and the second matching result has a second target feature corresponding to all of the semantic vectors; The video segment generation module generates a plurality of video segments suitable for the description of the beginning scene of the video, the description of the middle scene of the video, and the description of the ending scene of the video based on the first matching result and the second matching result; The video splicing module splices the multiple video clips based on the artistic expression technique information to obtain an initial short video; the artistic expression technique information includes sound effect and music information, visual symbol information and editing technique information; The video optimization module performs subtitle design and color tone conversion on the initial short video based on the element supplementary description information to obtain a target short video; the element supplementary description information includes subtitle design and color tone contrast information.

3. The method according to claim 2, characterized in that The semantic vectors include a semantic vector of a scene at the beginning of the video, a semantic vector of a scene in the middle of the video, and a semantic vector of a scene at the end of the video; The generating, based on the first matching result and the second matching result, a plurality of video clips applicable to the description of the beginning scene of the video, the description of the middle scene of the video, and the description of the ending scene of the video comprises: When the first matching result has a first target feature corresponding to all of the semantic vectors, acquiring material content corresponding to the first target feature corresponding to all of the semantic vectors from the temporary video material; Alternatively, when the first matching result has a first target feature corresponding to part of the semantic vector, obtain the first material content to be analyzed corresponding to the first target feature corresponding to part of the semantic vector from the temporary video material; determine the semantic vector to be analyzed that is not matched to the first material feature; determine the second target feature corresponding to the semantic vector to be analyzed from the second target features corresponding to all of the semantic vectors that exist in the second matching result; obtain the second material content to be analyzed corresponding to the second target feature corresponding to the semantic vector to be analyzed from the system material; and construct the material content corresponding to all of the semantic vectors based on the first material content to be analyzed and the second material content to be analyzed; Classify the above-mentioned material contents to obtain the material contents corresponding to the semantic vector of the scene at the beginning of the video, the material contents corresponding to the semantic vector of the scene in the middle of the video, and the material contents corresponding to the semantic vector of the scene at the end of the video; The material content corresponding to the semantic vector of the scene at the beginning of the video is used as a video clip suitable for describing the scene at the beginning of the video; the material content corresponding to the semantic vector of the scene in the middle of the video is used as a video clip suitable for describing the scene in the middle of the video; the material content corresponding to the semantic vector of the scene at the end of the video is used as a video clip suitable for describing the scene at the end of the video.

4. The method according to claim 3, characterized in that The constructing material content corresponding to all the semantic vectors according to the first material content to be analyzed and the second material content to be analyzed includes: Analyzing image features of the first material to be analyzed; The image feature is used as an algorithm parameter of a preset generation algorithm, and the image style of the second material content to be analyzed is adjusted by the preset generation algorithm to obtain an optimized second material content to be analyzed; the generation algorithm is a diffusion model; The first material content to be analyzed and the optimized second material content to be analyzed are merged into material content corresponding to all the semantic vectors.

5. The method according to claim 1, characterized in that: The generating of the target short video comprises: When the temporary video material is empty, classifying and marking the image frames of the system material to extract the second material feature of the system material; According to the semantic vector, material matching is performed on the second material feature to obtain a plurality of video clips suitable for describing the scene at the beginning of the video, describing the scene in the middle of the video, and describing the scene at the end of the video; Based on the artistic expression technique information, the multiple video clips are spliced ​​to obtain an initial short video; Based on the element supplementary description information, subtitles are designed and the color tone is converted for the initial short video to obtain a target short video.

6. The method according to claim 1, generating a pre-fine-tuned large language model according to the following steps, comprising: Obtaining video creation requirements and a preset video production solution knowledge base for a short video generation task, wherein the video creation requirements are used to characterize the basic requirements for creating a short video, and the preset video production solution knowledge base is constructed based on a standard data set related to short video creation, and the standard data set related to short video creation includes video scripts, scene description data, and emotion expression data; Use the pre-trained preset large language model as the basic model; According to the video creation requirements of the short video generation task, the basic model is modified to determine target parameters suitable for the video creation requirements; The basic model is fine-tuned and optimized through the sample data in the video production solution knowledge base and the target parameters to obtain a pre-fine-tuned large language model.

7. The method according to claim 6, characterized in that The method of fine-tuning and optimizing the basic model by using the sample data in the video production solution knowledge base and the target parameters to obtain a pre-fine-tuned large language model includes: Dividing the sample data in the video production solution knowledge base according to a preset ratio to obtain a first training set and a second training set; Decomposing the target parameters to obtain a first parameter decomposition matrix and a second parameter decomposition matrix, wherein the first parameter decomposition matrix includes parameters related to columns of the original model parameter matrix in the base model, and the second parameter decomposition matrix includes parameters related to rows of the original model parameter matrix in the base model; Calculating a back propagation result of the basic model according to each training data in the first training set, the first parameter decomposition matrix and the second parameter decomposition matrix; Calculating the model loss value of the basic model according to the back propagation result and the preset target loss function; When the loss value of the model reaches a minimum, a fine-tuned basic model is obtained; The second training set is used to perform secondary optimization on the fine-tuned basic model to obtain a pre-fine-tuned large language model.

8. The method according to claim 7, characterized in that The second training set is used to perform secondary optimization on the fine-tuned basic model, including: Inputting each training set in the second training set into the fine-tuned basic model, and outputting video production solution information corresponding to each training set; Analyze the performance quantification value of the fine-tuned basic model according to the video production solution information; Determine an evaluation value based on the performance quantification value and in combination with a preset evaluation function; In the case where the evaluation value does not reach the preset expected value, the parameters of the fine-tuned basic model are updated to maximize the expectation of the evaluation value determined by the preset evaluation function.

9. A short video automatic generation device based on a new media content creation model, characterized in that: The device comprises: A model calling module, used to call a preset new media content creation model upon receiving a short video generation request sent by a client; the short video generation request carries a video production description text and a user identifier, and the preset new media content creation model includes a pre-fine-tuned large language model and a video generation model; A text processing module, used for inputting the video production description text into a pre-fine-tuned large language model, and outputting video production plan information corresponding to the video production description text, wherein the video production plan information includes video script framework information, video artistic expression technique information, and element supplementary description information; A material calling module, used to search the temporary video material uploaded by the user and the system material related to the video production description text from the BI system through the user identifier; A short video generation module, configured to input the video production scheme information, the temporary video material and the system material into the video generation model to generate a target short video when the temporary video material is not empty; or, when the temporary video material is empty, input the video production scheme information and the system material into the video generation model to generate a target short video; The video sending module is used to output the target short video corresponding to the video production description text and send the target short video to the client.

Images