Video processing method and apparatus, and device and storage medium

Abstract

The embodiments of the present disclosure relate to a video processing method and apparatus, and a device and a storage medium. The method comprises: acquiring a requirement description text for making a video template, and determining first background information, which comprises an object label and the number of video clips of a video background; on the basis of the requirement description text, the first background information and candidate material information, generating constituent element information of the video template by using a generative model, wherein the constituent element information comprises at least one piece of music information, effect resource information and subtitle information; and generating a target video template on the basis of the constituent element information.

Description

Video processing method, device, equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed on October 18, 2023, with application number 202311354126.4 and invention name “Video processing method, device, equipment and storage medium”. The entire contents of that application are incorporated by reference into this application. Technical Field

[0003] The present disclosure relates to the field of video technology, and in particular to a video processing method, apparatus, device, and storage medium. Background Art

[0004] A video template is a fixed-format framework used to quickly create videos. Currently, video templates are primarily created manually using video template creation tools. This involves manually understanding the specific requirements and designing the necessary assets to create the template. However, this method is time-consuming and labor-intensive, and the quality of the video template is significantly affected by human effort, resulting in low production efficiency.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems, the embodiments of the present disclosure provide a video processing method, apparatus, device and storage medium.

[0007] In a first aspect, an embodiment of the present disclosure provides a video processing method, the method comprising:

[0008] Obtaining a text describing the requirements for producing a video template and determining first background information; wherein the first background information includes an object label of the video background and the number of video segments;

[0009] Based on the requirement description text, the first background information, and the candidate material information, using a generative model, generating component element information of the video template; wherein the component element information includes at least one of music information, effect resource information, and subtitle information;

[0010] Based on the component information, a target video template is generated.

[0011] In a second aspect, an embodiment of the present disclosure further provides a video processing device, the device comprising:

[0012] A requirement description text acquisition module is used to obtain a requirement description text for making a video template and determine first background information; wherein the first background information includes an object label of the video background and the number of video segments;

[0013] a component information generation module, which generates component information of a video template using a generative model based on the requirement description text, the first background information, and the candidate material information; wherein the component information includes at least one of music information, effect resource information, and subtitle information;

[0014] The video template generation module is used to generate a target video template based on the component element information.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0016] processor;

[0017] a memory for storing executable instructions;

[0018] The processor is used to read executable instructions from the memory and execute the executable instructions to implement the video processing method described in any embodiment of the present disclosure.

[0019] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the video processing method described in any embodiment of the present disclosure.

[0020] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which is used to execute the video processing method described in any embodiment of the present disclosure.

[0021] The video processing method, apparatus, device and storage medium of the embodiments of the present disclosure can obtain a demand description text for producing a video template and determine first background information including object labels of the video background and the number of video segments; based on the demand description text, the first background information and candidate material information, a generative model is used to generate constituent element information of the video template; wherein the constituent element information includes at least one of music information, effect resource information and subtitle information; and based on the constituent element information, a target video template is generated.

[0022] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0024] FIG1 is a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0025] FIG2 is a flow chart of another video processing method provided by an embodiment of the present disclosure;

[0026] FIG3 is a flow chart of another video processing method provided by an embodiment of the present disclosure;

[0027] FIG4 is a data interaction timing diagram of a video processing method provided by an embodiment of the present disclosure;

[0028] FIG5 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure;

[0029] FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0033] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0035] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0036] Video templates have been developed to help users create engaging videos more conveniently. They provide a video framework consisting of various elements needed to create a video, such as music, effects resources (also known as special effects), and subtitles. Users simply load their own video footage into a video template to create a composite video that incorporates the video effects specified in the template.

[0037] One approach to generating video templates is to provide an interactive video template creation tool. The creator designs the various elements to be included in the video template according to their needs. Through the interactive interface of the video template creation tool, the creator sets and inputs the designed elements one by one to generate the video template. This video template creation tool has the function of automatically generating video templates based on the designed elements. However, this approach relies heavily on manual labor, resulting in high production costs for the video templates. Furthermore, the quality of the video templates depends on the creator's skill, resulting in inconsistent quality. Another approach is to extract relevant element information from a reference video composed of multiple elements and use this extracted element information to generate a video template similar to the reference video. However, the video templates generated by this approach lack flexibility and cannot meet the personalized needs of users.

[0038] Based on the above situation, the embodiment of the present disclosure provides a video processing solution, which calls a pre-trained generative model according to the user's requirements for video template production and the determined first background information of the video template, generates a video template that meets the user's needs, reduces the manual participation and time cost of video template generation, and improves the generation efficiency, generation flexibility and quality of the video template.

[0039] The video processing method provided in the embodiments of the present disclosure is applicable to scenarios where video templates are created or where a composite video is generated based on a video template. The method can be performed by a video processing device, which can be implemented in software and / or hardware. The device can be integrated into an electronic device with human-computer interaction capabilities and a certain level of computing power, such as, but not limited to, a smartphone, a tablet personal computer (Tablet PC), a laptop computer, a desktop computer, a server, and the like.

[0040] FIG1 shows a flow chart of a video processing method provided by an embodiment of the present disclosure. As shown in FIG1 , the video processing method may include the following steps:

[0041] S110 , obtaining a text describing requirements for producing a video template, and determining first background information, where the first background information includes an object label of the video background and the number of video segments.

[0042] The requirement description text is a text that describes the requirements of the video template to be produced. It can be a sentence or a paragraph. For example, the requirement description text can be "I need to generate a romantic and beautiful wedding music video type video template." The first background information is related information describing the video background in the video template. It includes at least the number of segments to be divided into for the entire video (i.e., the number of video segments) and the identification information of the objects contained in the video background (e.g., sky, clouds, mountains, roads, cars, etc.) (i.e., object labels, such as names or pre-set numbers, etc.).

[0043] Specifically, in order to reduce the complexity and cost of video template production, an interactive interface can be provided to the user to receive information input by the user that can express the requirements of the video template to be produced. The interactive interface can be an interactive interface provided by a video processing device, for example, it can be an interactive interface provided by secondary packaging of a video template production tool in the related art, or it can be an interactive interface of a new application. Then, the input demand information is processed to obtain a demand description text for producing the video template. For example, if the demand information input by the user is voice, the voice is converted into text to obtain a demand description text; if the demand information input by the user is text, it can be directly used as a demand description text; if the above-mentioned text is not concise enough, the obtained text can be further extracted and modified to obtain a more concise text that can fully express the user's demand for the production of the video template as the demand description text.

[0044] In addition, to further improve the fit between subsequently generated video templates and user needs, first background information can be provided as reference information for video template creation. This first background information can be obtained based on user input or extracted based on user interaction data for videos or video templates containing at least one video element within a historical time period.

[0045] In some embodiments, determining the first context information may be implemented by: in response to a triggering operation on a context information input control, determining the input object label and the number of video segments as the first context information.

[0046] Specifically, the electronic device may provide an input control for inputting background information (i.e., a background information input control) in the interactive interface. After the user triggers the input control and inputs content (such as text or voice), the electronic device may obtain the first background information based on the user's input content.

[0047] In other embodiments, determining the first context information may be implemented as: in response to a triggering operation on a video input control, performing video recognition on an input first reference video to generate the first context information.

[0048] Specifically, the electronic device may provide an input control (i.e., a video input control) in the interactive interface for inputting video materials with reference significance. The user may input the video material as the first reference video by triggering the input control. Then, the electronic device may call the video recognition model to identify and extract the video content of the first reference video to obtain the first background information. For example, for the first reference video of a wedding dress, after video recognition, the first background information can be obtained: {"slots":4,"tags":["sea","reef","sky","wedding dress","holding hands","cheongsam","wedding photo"]}. The video recognition model here is not limited to a specific model form, as long as it can realize the recognition of the video content.

[0049] In other embodiments, determining the first background information may be implemented as follows: in response to a triggering operation on a style input control, based on the input video style information, screening a second reference video from candidate videos, and performing video recognition on the second reference video to generate the first background information.

[0050] Among them, video style can be understood as the form of expressing video content. For example, the video style can be a real-life voice-over background style, a narrative narration style, a story theater style, a production process display style, and a creative expression style. Video style information is information that expresses the style of the video, such as the name or number of the video style, a brief description of the video style form, etc. Candidate videos are multiple video materials pre-collected in the database. The database can be stored locally, in the cloud, or on a server, in which candidate videos and their corresponding video style information can be pre-stored. The second reference video is a video material selected from the candidate videos for reference when making a video template.

[0051] Specifically, the electronic device may provide an input control (i.e., a style input control) in the interactive interface for inputting a video style. The user can trigger this input control to input the video style information of the video template they wish to create. The electronic device can then match the input video style information with various video style information stored in a database and determine the candidate video corresponding to the stored video style information that successfully matches as the second reference video. The electronic device can then identify and extract the video content of the second reference video to obtain the first background information.

[0052] In some further embodiments, determining the first context information may be implemented by screening a second reference video from candidate videos based on historical behavior data, and performing video recognition on the second reference video to generate the first context information.

[0053] Among them, historical behavior data is the user's behavior data in a historical time period, which at least includes interaction data (such as browsing, sharing, liking, collecting, etc.) on videos or video templates containing at least one video element.

[0054] Specifically, a user's historical behavior data can, to a certain extent, reflect their preferred video style. Therefore, if the user does not actively input any of the first context information, the first reference video, or the video style information, the electronic device can use the user's historical behavior data to determine the user's preferred video style information. This video style information is then used to obtain a second reference video, and the first context information is then obtained by identifying the content of the second reference video.

[0055] S120 : Based on the demand description text, the first background information, and the candidate material information, the component element information of the video template is generated using a generative model.

[0056] Among them, the candidate material information is a plurality of materials (i.e., candidate materials) collected in advance and their related information, such as the candidate material number, name, description information describing its content or effect, display style, applicable scenarios or applicable industries, etc., at least one of which. A generative model is a pre-trained machine learning model that can generate other output information based on some input information. The form of the model is not limited, as long as it can output the constituent element information of the video template. It is understandable that the larger the model scale of the generative model, the more accurate the constituent element information it outputs. The constituent element information is the information of the various elements that constitute the video template. The elements here can be, for example, at least one of the music as the soundtrack in the video template, the effect resources as the specific effects of the display video (such as stickers, transition animation effects, filter effects, etc.), and the subtitles (such as descriptive or explanatory subtitles, titles, etc.) that display the relevant content in the video.

[0057] Specifically, the electronic device can organize the demand description text, the first background information and the candidate material information into the input form of the generative model, and input the organized content into the generative model, so that the generative model can screen out the materials that meet the requirements from the candidate materials based on the demand description text and the first background information, and determine the screened material information as the constituent element information for output. In this way, the constituent element information for making a video template can be obtained through a small amount of demand description text input by the user, such as: the video template is divided into 4 segments, each segment is configured with music of xx information, stickers of xx information, filter effects of xx information, transition effects of xx information are configured between each segment, and xx time period of the video is configured with xx subtitles, etc. In this way, the process of manually designing the video template and manually determining the constituent element information of the video template is omitted, which greatly reduces the degree of manual participation and the difficulty of making the video template, and improves the efficiency and flexibility of making the video template.

[0058] S130: Generate a target video template based on the component element information.

[0059] Specifically, the electronic device can request the corresponding constituent element material content (such as music material, effect resource material, etc.) from the database based on the obtained constituent element information, and then summarize the various material contents according to the preset data structure corresponding to the video template to obtain the final video template, that is, the target video template.

[0060] In some embodiments, S130 can be implemented as follows: based on the constituent element information, calling the preset operation function in the preset instruction interface to convert the constituent element information into an operation instruction of the video template production tool; based on the operation instruction, calling the video template production tool to generate the target video template.

[0061] Among them, the preset instruction interface (Application Programming Interface, API) is another interactive interface defined by pre-encapsulating the human-computer interaction interface of the video template production tool, which is used to receive the results of the generative model output and convert it into instructions that can be received by the video template production tool. For example, the preset instruction interface can be defined as interface***API{}, and "***API" is the interface name of the preset instruction interface. The preset operation function is a functional function defined in the preset instruction interface, which corresponds to the constituent elements of the video template. For example, for music elements, the preset operation function can be specifically implemented as a functional function that can input music elements into the video template production tool to add the music elements to the video template. It can be exemplified by fun addMusic(id:String), "addMusic()" is the function name of the preset operation function, and "id:String" is the parameter and its data type that the defined function needs to input.

[0062] Specifically, in order to automatically input the component element information output by the generative model into the video template production tool, corresponding preset operation functions can be defined according to the information input requirements of the interactive interface for each component element information provided by the video template production tool, and then each preset operation function can be encapsulated as a functional function in a preset instruction interface.

[0063] In this way, after obtaining the information of each constituent element, the electronic device can call the corresponding preset operation function in the preset instruction interface. For example, calling the music element operation function with the element number id "7119417156021xxxxxx" can be implemented as ***API.addMusic("7119417156021xxxxxx"). Through the processing of the preset operation function, the constituent element information output by the generative model is converted into the operation instruction of the video template production tool to trigger the video template production tool to implement the corresponding video production function. For example, for music elements, video production functions such as inputting music element information, requesting music elements from the database, and summarizing the obtained music elements according to the preset data structure of the video template can be implemented. For other constituent elements, the same video production function can be implemented to obtain the target video template.

[0064] The above settings can use preset instruction interfaces and preset operation functions to replace the manual operation process of using video template production tools to set and input the constituent element information of the video template, further improving the degree of automation of video template production, thereby further reducing the cost of video template production and further improving the efficiency of video template generation.

[0065] In some embodiments, S130 may be implemented as: generating a target video template based on the component information and the second background information.

[0066] Among them, the second background information is another relevant information describing the video background in the video template, which at least includes the video background content and video filler clips of each segmented video. The video background content of the segmented video can be obtained by the video content of the corresponding segment identified in the target reference video (segmented video content for short), which can be the video part of the identified segment or the picture in the video part of the identified segment. The video filler clip is a time period in the video template that can be inserted into the video material input by the user for synthesizing the final synthetic video, which can be determined by identifying the content of the target reference video. The target reference video here can be the first reference video or the second reference video.

[0067] Specifically, the component information output by the generative model includes information about components involved in the video template, such as music elements, special effects elements, and letter elements. This information can be used to generate a target video template, but such a video template is relatively less interesting. Therefore, in this embodiment, the component information can be superimposed with the second background information to increase the video background content in the video template, while also providing video filler clips that can be inserted into the user's video material. In this way, the target video template obtained by calling the video target creation tool contains both the component elements and the video background content and video filler clips, further enhancing the content richness and interest of the target video template.

[0068] The video processing method provided by the above-mentioned embodiments of the present disclosure can obtain the demand description text for making a video template, and determine the first background information including the object label of the video background and the number of video segments; based on the demand description text, the first background information and the candidate material information, the generative model is used to generate the constituent element information of the video template; wherein the constituent element information includes at least one of music information, effect resource information and subtitle information; based on the constituent element information, the target video template is generated; the user only needs to input a small amount of information such as the demand description text to automatically generate the required video template, which greatly reduces the labor cost and time cost in the video template production process, improves the production efficiency of the video target, and adopts a pre-trained generative model, which can greatly reduce the dependence of the video template on human experience and improve the quality of the video template.

[0069] In some embodiments, after generating the target video template in S130, the video processing method further includes: in response to a triggering operation on a template editing control, modifying component information of the target video template to update the target video template.

[0070] Specifically, the target video template generated by the electronic device through the above-described embodiments is automatically generated based on the requirement description text, and may contain portions that do not meet the user's needs. Therefore, the user can be provided with a function for secondary editing of the target video template, namely, a template editing control is provided in the interactive interface. When the user triggers the template editing control, the component information in the target video template can be modified and edited through the interactive interface. The electronic device can then regenerate the video template based on the user's modification results, thus updating the target video template.

[0071] It should be noted that when the target video template includes the second background information, the user can also modify / adjust the second background information through the template editing control and its corresponding interactive interface to update the target video template.

[0072] FIG2 is a flow chart of another video processing method provided by an embodiment of the present disclosure. This embodiment adds a step of "generating a composite video" based on the above embodiments. Explanations of terms / steps that are identical or similar to those in the above embodiments are not repeated here. Referring to FIG2 , the video processing method provided by this embodiment includes:

[0073] S210: Obtain a text describing the requirements for producing a video template and determine first background information.

[0074] S220: Based on the demand description text, the first background information and the candidate material information, the component element information of the video template is generated using a generative model.

[0075] S230: Generate a target video template based on the component element information.

[0076] S240 : In response to the video generation operation, generate a composite video based on the target video template and the input video material.

[0077] Specifically, in the related art, when using the video material input by the user to generate a composite video, it is necessary to match the most suitable video template from a variety of video templates, and then load the user's video material. This process consumes a large amount of computing resources, resulting in low efficiency in generating the composite video. In the embodiment of the present disclosure, the demand description text input by the user can be used to generate a target video template that meets the user's needs, and then the video material input by the user for making the composite video is loaded into the target video template, replacing the video filler clips in the target video template to generate the composite video. This can greatly reduce the resource consumption in the composite video generation process, and can make the composite video more in line with the user's video needs, thereby improving the generation efficiency and quality of the composite video.

[0078] FIG3 is a flow chart of another video processing method provided by an embodiment of the present disclosure. Based on the above embodiments, this embodiment further optimizes the "generation of component element information of the video template based on the demand description text, the first background information, and the candidate material information using the generative model." The explanation of the same or similar terms / steps as those in the above embodiments will not be repeated here. Referring to FIG3, the video processing method provided by this embodiment includes:

[0079] S310: Obtain a text describing the requirements for producing a video template and determine first background information.

[0080] S320: Based on the requirement description text, the first background information, and the candidate material information, an input prompt text of the generative model is constructed according to a preset input template corresponding to the generative model.

[0081] The preset input template is a text template of a prompt instruction that is pre-defined for a video template production scenario, and specifies a text framework of the prompt instruction in the video template production scenario.

[0082] Specifically, a pre-trained generative model learns a lot of knowledge during the pre-training process. To standardize the generative model and make its output more suitable for the video template creation scenario, the generative model can be customized with input prompts corresponding to the video template creation scenario. In the disclosed embodiment, the requirement description text, the first background information, and the candidate material information can be processed according to a preset input template to generate the input prompt text.

[0083] In some embodiments, S330 may be implemented as the following steps A and B.

[0084] Step A: Process the requirement description text, the first background information and the candidate material information according to preset data formats to generate a requirement format text, a background format text and a material format text.

[0085] Specifically, the preset input template can specify the data format of part of the text involved in the prompt instruction (i.e., the preset data format). Exemplarily, the preset data format can be set to a data format with cross-platform parsing capabilities and good readability, such as a JS object notation data format, an extensible markup language data format, etc. In this way, the requirement description text, the first background information, and the candidate material information can be sorted separately according to the preset data format to obtain the corresponding data stored in the preset data format, i.e., the requirement format text, the background format text, and the material format text.

[0086] It should be noted that the number of various types of element information contained in the candidate material information and its corresponding material format text in the embodiment of the present disclosure can vary, and the specific information dimensions can also be increased or decreased, etc., which are not specifically limited in the embodiment of the present disclosure.

[0087] Step B: Use the required format text, background format text, and material format text to replace the placeholders in the preset input template and construct the input prompt text.

[0088] Specifically, the preset input template can define the text frame of the prompt instruction, which includes some fixed text content and some replaceable text placeholders. An example is as follows:

[0089] val promptFormat="You are an AI video editing expert. I will provide you with *** information. Based on the number of video clips (slots), *** information, and video template requirements, (e.g., %1$s) please help me select appropriate special effects resources, name the video (<= n characters), and describe it (<= m characters). The selection of various special effects must be based on a professional perspective of video editing and synthesis, and must be consistent with the video scene, the theme, and the requirements of the video template. Do not use arbitrary or rote effects. You must strictly follow the following example return. No further explanation is needed. The number of filter effects selected should be <= the number of slots, and the number of transition effects selected should be slots - 1. If there are not enough transition effect resources, duplicate transition effects are allowed. The return example is %2$s. Note that the returned special effects resources must be in the ** data format. The resources you need to refer to are %3$s."

[0090] In the above example, "%1$s" corresponds to the text placeholders for the required format text and background format text. "%3$s" corresponds to the text placeholder for the source format text. "%2$s" corresponds to the text placeholder for the sample text in the data format output by the model (referred to as output format text). The sample data format output by the model can be customized in advance based on business needs. In the above example, n and m are the predefined word counts for the video title and video description, respectively.

[0091] The electronic device can replace the above text placeholders in the preset input template with the above-obtained demand format text, background format text, material format text and output format text respectively, and thus obtain the input prompt text for the target video template production.

[0092] In some embodiments, S330 can be implemented as follows: if the constituent element information includes subtitle information, then in response to the triggering operation of the subtitle control, the subtitle operation result is determined; based on the demand description text, the first background information, the candidate material information and the subtitle operation result, an input prompt text is constructed according to a preset input template.

[0093] The subtitle operation result refers to the result generated after the user performs the corresponding interactive operation on the subtitle-related function in the video template. For example, if the interactive operation performed is to enter subtitle content, the subtitle operation result is based on the entered subtitle content; if the interactive operation performed is to check the option corresponding to the pre-set subtitle-related content, the subtitle-related content of the checked option is determined as the subtitle operation result, etc.

[0094] Specifically, the video template may include subtitle information. In order to further improve the flexibility of the video template generation and its fit with user needs, the subtitle information is not set as fixed content, but can be input by the user or generated by a generative model. Accordingly, in order to achieve these functions, corresponding interactive controls, namely subtitle controls, can be set in the interactive interface. The user can determine whether to add subtitle information to the target video template by whether to perform a trigger operation on the subtitle control. After the user performs a trigger operation on the subtitle control, the electronic device can use the subtitle operation result as one of the input contents of the generative model to construct the input prompt text. The processing process of the subtitle operation result can be found in the description of step A above.

[0095] In one example, the above-mentioned determination of the subtitle operation result in response to the triggering operation of the subtitle control can be implemented as: in response to the check operation of the subtitle option, determining the preset prompt content as the subtitle operation result.

[0096] Specifically, the instructional text corresponding to the subtitles, i.e., preset prompt content, can be pre-defined, and a placeholder can be set for it in the preset input template. The user can also be provided with an interactive control, i.e., a subtitle option, to determine whether to add the preset prompt content to the preset input template. For example, a checkbox can be displayed in the interactive interface to indicate whether to generate subtitle content such as the video title / description content / video template design concept. If the user checks the subtitle option, the electronic device can use the preset prompt content as the result of the subtitle operation and replace the corresponding placeholder in the preset input template in the subsequent process.

[0097] For example, the preset prompt content may be the underlined text content in the preset input template in the above example, such as "Name the video (<=n words), Describe the video (<=m words)". After the user selects a subtitle option, the preset prompt content corresponding to the selected subtitle option may be determined as the subtitle operation result.

[0098] In another example, the above-mentioned determination of the subtitle operation result in response to the triggering operation of the subtitle control can be implemented as follows: in response to the triggering operation of the subtitle input control, the input subtitle content is processed according to a preset data format, and subtitle format text is generated as the subtitle operation result.

[0099] Specifically, the electronic device may also provide an input control (i.e., a subtitle input control) in the interactive interface for inputting subtitle content. The user can trigger the subtitle input control and input the subtitle content they want to display in the video template in the form of text or voice. The electronic device stores the text of the subtitle content input by the user in a preset data format, and obtains subtitle formatted text as the result of the subtitle operation.

[0100] Based on the embodiment in which the component information includes subtitle information, step B can be implemented as follows: using the required format text, background format text, material format text, and the subtitle operation result to replace the placeholders in the preset input template, thereby constructing the input prompt text. That is, the electronic device uses the required format text, background format text, material format text, and the subtitle operation result to replace the corresponding placeholders in the preset input template, thereby constructing the final input prompt text prompt.

[0101] S330: Based on the input prompt text, call the generative model to generate component element information.

[0102] Specifically, the electronic device may set the input prompt text obtained above as the prompt instruction prompt of the generative model, and call the generative model to obtain the constituent element information of the video template.

[0103] It should be noted that if the above-mentioned input prompt text is too long and exceeds the upper limit of the prompt instruction input of the generative model, the input prompt text can be split into shorter prompt texts, and the generative model can be called multiple times in a related step-by-step manner to obtain the constituent element information of the video template.

[0104] S340: Generate a target video template based on the component element information.

[0105] Based on the above examples, the component element information ultimately output by the generative model may include the ID of at least one music element, the ID of at least one sticker element, the ID of at least one filter effect, the ID of at least one transition effect, and may also include video title content, video description content, and so on. The video template creation tool can request the corresponding component materials based on this component element information and organize the obtained component materials, video title content, video description content, etc. into a target video template according to the preset data structure of the video template.

[0106] The video processing method provided by the above-mentioned embodiments of the present disclosure can adaptively process the demand description text, the first background information and the candidate material information according to the preset input template corresponding to the generative model, and construct the input prompt text of the generative model; then use the input prompt text to call the generative model to generate the constituent element information of the video template; realize the use of the user's demand description text, the first background information and the candidate material information to obtain the constraint prompt text prompt of the generative model, further strengthen the output result of the generative model, make the generated video template more consistent with the user's needs, thereby further improving the generation efficiency and quality of the video template.

[0107] FIG4 is a data interaction sequence diagram of a video processing method provided by an embodiment of the present disclosure. This embodiment is based on the above embodiments, and the same or similar terms / steps contained therein can be found in the descriptions of the above embodiments. Referring to FIG4 , the electronic device may include a video template creation tool, a video recognition function module, and a generative model function module. The data interaction process of the video processing method includes the following steps:

[0108] S401: The user (or video template creator) enters a description of the requirements and a first reference video and / or video style information through the interactive interface of the video template creation tool. If the first reference video is entered, S403 is executed; if the video style information is entered, S402 is executed. If both the first reference video and video style information are entered, both S402 and S403 are executed.

[0109] S402: The video template creation tool sends the video style information to the server, so that the server can filter and obtain a second reference video from the database according to the video style information.

[0110] S403: The video recognition function module receives the first reference video and / or the second reference video, performs video recognition on the video, obtains the first background information and the second background information, and sends the information to the video template creation tool.

[0111] If the video recognition function module obtains the first reference video and the second reference video at the same time, one implementation method is: perform video recognition on the first reference video to ensure that the first background information and the second background information are more in line with user needs; another implementation method is: perform video recognition on both the first reference video and the second reference video, and use the recognition result of the second reference video to correct the recognition result of the first reference video. For example, a larger weight can be set for the first reference video and a smaller weight can be set for the second reference video, and then the recognition results of the two can be weighted calculated to obtain the final first background information and second background information.

[0112] S404: The video template creation tool constructs an input prompt text based on the requirement description text, the first background information, and the candidate material information.

[0113] S405. The generative model function module receives the input prompt text, runs the generative model based on it, and outputs the component element information of the video template.

[0114] S406: The video template making tool receives the component element information of the video template sent by the generative model function module, and converts the component element information into an operation instruction that can be received by the video template making tool.

[0115] S407 , the video template making tool sends a request to the server to obtain the constituent element materials according to the operation instruction, receives the constituent element materials fed back by the server, and generates a target video template using the constituent element materials, constituent element information and the second background information.

[0116] S408: The video template creation tool modifies the component information of the target video template in response to the triggering operation on the template editing control to update the target video template.

[0117] S409 , the video template making tool receives the video material input by the user in response to the video generating operation, and generates a composite video based on the target video template and the input video material.

[0118] The following is an embodiment of a video processing device provided by an embodiment of the present invention. The device and the video processing methods of the above embodiments belong to the same inventive concept. For details not fully described in the embodiment of the video processing device, please refer to the embodiment of the above video processing method.

[0119] FIG5 shows a schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure. As shown in FIG5 , the video processing device 500 may include:

[0120] The requirement description text acquisition module 510 is used to obtain the requirement description text for making a video template and determine the first background information; wherein the first background information includes the object label of the video background and the number of video segments;

[0121] The component information generation module 520 generates component information of the video template using a generative model based on the requirement description text, the first background information, and the candidate material information; wherein the component information includes at least one of music information, effect resource information, and subtitle information;

[0122] The video template generation module 530 is configured to generate a target video template based on the component information.

[0123] In some embodiments, the requirement description text acquisition module 510 is specifically configured to determine the first background information by any of the following methods:

[0124] In response to a triggering operation on a video input control, performing video recognition on an input first reference video to generate first background information;

[0125] In response to a triggering operation on the style input control, based on the input video style information, a second reference video is screened from the candidate videos, and video recognition is performed on the second reference video to generate first background information;

[0126] Based on the historical behavior data, a second reference video is screened from the candidate videos, and video recognition is performed on the second reference video to generate first background information.

[0127] In some embodiments, the component information generation module 520 includes:

[0128] An input prompt text construction submodule is used to construct an input prompt text of the generative model based on the requirement description text, the first background information and the candidate material information according to a preset input template corresponding to the generative model;

[0129] The component information generation module is used to call the generative model based on the input prompt text to generate component information.

[0130] Furthermore, the input prompt text construction submodule is specifically used to:

[0131] Processing the requirement description text, the first background information, and the candidate material information according to preset data formats to generate a requirement format text, a background format text, and a material format text;

[0132] Use the required format text, background format text and material format text to replace the placeholders in the preset input template and construct the input prompt text.

[0133] Optionally, the input prompt text construction submodule includes:

[0134] a subtitle operation result determining unit, configured to determine a subtitle operation result in response to a triggering operation on a subtitle control if the constituent element information includes subtitle information;

[0135] The input prompt text construction unit is used to construct the input prompt text according to a preset input template based on the demand description text, the first background information, the candidate material information and the subtitle operation result.

[0136] Furthermore, the subtitle operation result determination unit is specifically configured to:

[0137] In response to a check operation on a subtitle option, determining a preset prompt content as a subtitle operation result;

[0138] Alternatively, in response to a triggering operation on a subtitle input control, the input subtitle content is processed according to a preset data format to generate subtitle format text as a subtitle operation result.

[0139] In some embodiments, the video template generation module 530 is specifically configured to:

[0140] A target video template is generated based on the constituent element information and the second background information; wherein the second background information includes segmented video content and video filling clips in the target reference video; the target reference video includes the first reference video or the second reference video, and the video filling clips are used to fill the input video material.

[0141] In some embodiments, the video template generation module 530 is further specifically configured to:

[0142] Based on the component information, the preset operation function in the preset instruction interface is called to convert the component information into an operation instruction of the video template making tool;

[0143] Based on the operation instructions, the video template making tool is called to generate the target video template.

[0144] In some embodiments, the video processing apparatus 500 further includes a video template updating module configured to:

[0145] After the target video template is generated based on the component information, the component information of the target video template is modified in response to a triggering operation on the template editing control to update the target video template.

[0146] In some embodiments, the video processing apparatus 500 further includes a composite video generation module configured to:

[0147] In response to the video generation operation, a composite video is generated based on the target video template and the input video material.

[0148] The video processing device provided in the embodiment of the present invention can execute the video processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0149] It is worth noting that in the embodiment of the above-mentioned video processing device, the various modules, sub-modules and units included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional modules / sub-modules / units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this disclosure.

[0150] The present disclosure also provides an electronic device that may include a processor and a memory, wherein the memory may be used to store executable instructions, wherein the processor may be used to read the executable instructions from the memory and execute the executable instructions to implement the video processing method in the above embodiment.

[0151] FIG6 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

[0152] As shown in Figure 6, the electronic device 600 may include a processing device 601 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output interface (I / O interface) 605 is also connected to the bus 604.

[0153] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data.

[0154] It should be noted that the electronic device 600 shown in FIG6 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure. That is, although FIG6 shows an electronic device 600 having various devices, it should be understood that it is not required to implement or include all of the devices shown. More or fewer devices may be implemented or included instead.

[0155] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the video processing method of any embodiment of the present disclosure are performed.

[0156] An embodiment of the present disclosure further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the video processing method in any embodiment of the present disclosure.

[0157] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0158] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP, and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0159] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0160] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the steps of the video processing method described in any embodiment of the present disclosure.

[0161] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0164] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0166] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0167] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A video processing method, comprising: Obtaining a text describing the requirements for making a video template and determining first background information; wherein the first background information includes an object label of the video background and the number of video segments; Based on the requirement description text, the first background information and the candidate material information, using a generative model, generating component information of the video template; wherein the component information includes at least one of music information, effect resource information and subtitle information; Based on the component information, a target video template is generated.

2. The method according to claim 1, wherein the determining the first background information comprises any one of the following: In response to a trigger operation on a video input control, performing video recognition on an input first reference video to generate the first background information; In response to a trigger operation on a style input control, based on the input video style information, a second reference video is selected from candidate videos, and video recognition is performed on the second reference video to generate the first background information; Based on the historical behavior data, the second reference video is screened from the candidate videos, and video recognition is performed on the second reference video to generate the first background information.

3. The method according to claim 1, wherein the step of generating the component element information of the video template based on the requirement description text, the first background information and the candidate material information by using a generative model comprises: Based on the requirement description text, the first background information and the candidate material information, constructing an input prompt text of the generative model according to a preset input template corresponding to the generative model; Based on the input prompt text, the generative model is called to generate the component element information.

4. The method according to claim 3, wherein the step of constructing the input prompt text of the generative model based on the requirement description text, the first background information and the candidate material information according to a preset input template corresponding to the generative model comprises: Processing the requirement description text, the first background information and the candidate material information according to preset data formats respectively to generate a requirement format text, a background format text and a material format text; The demand format text, the background format text and the material format text are used to replace the placeholders in the preset input template respectively to construct the input prompt text.

5. The method according to claim 3, wherein the step of constructing the input prompt text of the generative model based on the requirement description text, the first background information and the candidate material information according to a preset input template corresponding to the generative model comprises: If the component information includes the subtitle information, determining a subtitle operation result in response to a trigger operation on a subtitle control; Based on the requirement description text, the first background information, the candidate material information and the subtitle operation result, the input prompt text is constructed according to the preset input template.

6. The method according to claim 5, wherein the determining the subtitle operation result in response to the triggering operation on the subtitle control comprises: In response to a check operation on a subtitle option, determining a preset prompt content as the subtitle operation result; Alternatively, in response to a triggering operation on a subtitle input control, the input subtitle content is processed according to a preset data format to generate subtitle format text as the subtitle operation result.

7. The method according to claim 2, wherein generating a target video template based on the component information comprises: The target video template is generated based on the constituent element information and the second background information; wherein the second background information includes segmented video content and video filling fragments in the target reference video; the target reference video includes the first reference video or the second reference video, and the video filling fragments are used to fill the input video material.

8. The method according to any one of claims 1 to 7, wherein generating a target video template based on the component element information comprises: Based on the component element information, calling a preset operation function in a preset instruction interface to convert the component element information into an operation instruction of a video template making tool; Based on the operation instruction, the video template making tool is called to generate the target video template.

9. The method according to claim 2, wherein after generating a target video template based on the component information, the method further comprises: In response to a triggering operation on a template editing control, the constituent element information of the target video template is modified to update the target video template.

10. The method according to claim 1 or 9, wherein the method further comprises: In response to a video generation operation, a composite video is generated based on the target video template and the input video material.

11. A video processing device, comprising: A requirement description text acquisition module is used to acquire the requirement description text for making a video template and determine the first background information; wherein the first background information includes the object label of the video background and the number of video segments; A component information generation module, based on the requirement description text, the first background information and the candidate material information, generates component information of the video template using a generative model; wherein the component information includes at least one of music information, effect resource information and subtitle information; The video template generation module is used to generate a target video template based on the component element information.

12. An electronic device comprising: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to: Obtaining a text describing the requirements for making a video template and determining first background information; wherein the first background information includes an object label of the video background and the number of video segments; Based on the requirement description text, the first background information and the candidate material information, using a generative model, generating component information of the video template; wherein the component information includes at least one of music information, effect resource information and subtitle information; Based on the component information, a target video template is generated.

13. The electronic device according to claim 12, wherein the determining the first context information comprises any one of the following: In response to a trigger operation on a video input control, performing video recognition on an input first reference video to generate the first background information; In response to a trigger operation on a style input control, based on the input video style information, a second reference video is selected from candidate videos, and video recognition is performed on the second reference video to generate the first background information; Based on the historical behavior data, the second reference video is screened from the candidate videos, and video recognition is performed on the second reference video to generate the first background information.

14. The electronic device according to claim 12, wherein the generating component information of the video template using a generative model based on the requirement description text, the first background information and the candidate material information comprises: Based on the requirement description text, the first background information and the candidate material information, constructing an input prompt text of the generative model according to a preset input template corresponding to the generative model; Based on the input prompt text, the generative model is called to generate the component element information.

15. The electronic device according to claim 14, wherein the step of constructing the input prompt text of the generative model based on the requirement description text, the first background information and the candidate material information according to a preset input template corresponding to the generative model comprises: Processing the requirement description text, the first background information and the candidate material information according to preset data formats respectively to generate a requirement format text, a background format text and a material format text; The demand format text, the background format text and the material format text are used to replace the placeholders in the preset input template respectively to construct the input prompt text.

16. The electronic device according to claim 14, wherein the step of constructing the input prompt text of the generative model based on the requirement description text, the first background information and the candidate material information according to a preset input template corresponding to the generative model comprises: If the component information includes the subtitle information, determining a subtitle operation result in response to a trigger operation on a subtitle control; Based on the requirement description text, the first background information, the candidate material information and the subtitle operation result, the input prompt text is constructed according to the preset input template.

17. The electronic device according to claim 16, wherein the determining the subtitle operation result in response to the triggering operation on the subtitle control comprises: In response to a check operation on a subtitle option, determining a preset prompt content as the subtitle operation result; Alternatively, in response to a triggering operation on a subtitle input control, the input subtitle content is processed according to a preset data format to generate subtitle format text as the subtitle operation result.

18. The electronic device according to claim 13, wherein generating a target video template based on the component information comprises: The target video template is generated based on the constituent element information and the second background information; wherein the second background information includes segmented video content and video filling fragments in the target reference video; the target reference video includes the first reference video or the second reference video, and the video filling fragments are used to fill the input video material.

19. The electronic device according to any one of claims 12 to 18, wherein generating a target video template based on the component element information comprises: Based on the component element information, calling a preset operation function in a preset instruction interface to convert the component element information into an operation instruction of a video template making tool; Based on the operation instruction, the video template making tool is called to generate the target video template.

20. A computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the video processing method according to any one of claims 1 to 10.

Images