Video generation method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a video generation method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a plurality of video clips for a target object and object information of the target object, wherein the plurality of video clips respectively have corresponding first identification information; using a trained machine learning model to generate a target file based on the plurality of video clips and cue words, the cue words at least comprising object information and respective first identification information of the plurality of video clips, the target file at least comprises first identification information corresponding to at least one video clip arranged in sequence in the plurality of video clips, text description of the at least one video clip and a segmentation result of the text description; and generating a target video for the target object at least based on the target file.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly to methods, devices, apparatuses, computer-readable storage media, and computer program products for video generation. Background Art

[0002] Video editing technology has made rapid progress recently, and there are a variety of video editing methods. However, many video editing methods are inefficient and the quality of the edited content is low. Therefore, there is a need for a video editing method that is efficient and can generate high-quality films. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for video generation is provided. The method includes: obtaining multiple video clips for a target object and object information of the target object, wherein the multiple video clips respectively have corresponding first identification information; using a trained machine learning model, generating a target file based on the multiple video clips and prompt words, wherein the prompt words at least include object information and first identification information of each of the multiple video clips, and the target file at least includes the first identification information corresponding to at least one video clip arranged in sequence among the multiple video clips, a text description of the at least one video clip, and a segmentation result of the text description; and generating a target video for the target object based at least on the target file.

[0004] In a second aspect of the present disclosure, a device for video generation is provided. The device includes: an acquisition module configured to acquire multiple video clips for a target object and object information of the target object, wherein the multiple video clips respectively have corresponding first identification information; a file generation module configured to generate a target file based on the multiple video clips and prompt words using a trained machine learning model, wherein the prompt words at least include object information and first identification information of each of the multiple video clips, and the target file at least includes the first identification information corresponding to at least one video clip arranged in sequence among the multiple video clips, a text description of the at least one video clip, and a segmentation result of the text description; and a video generation module configured to generate a target video for the target object based at least on the target file.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1A A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 1B A schematic diagram showing another example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 A schematic diagram showing an example architecture for video generation at the inference stage according to some embodiments of the present disclosure;

[0013] Figure 3 A flowchart showing a process of machine learning model training according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A flowchart showing a process for machine learning model evaluation during a testing phase according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A flowchart of a method for video generation according to some embodiments of the present disclosure is shown;

[0016] Figure 6 An exemplary structural block diagram of an apparatus for video generation according to some embodiments of the present disclosure is shown; and

[0017] Figure 7 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0026] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0027] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0028] Figure 1AA schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the environment 100, an electronic device 110 applies a machine learning model 105 to perform video editing. The machine learning model 105 is configured to generate a target file 114. The machine learning model 105 is configured to process a prompt word 112 input by a user to generate the target file 114. The prompt word 112 can be a text prompt word to describe the task to be performed. The machine learning model 105 can be configured to perform a video editing task, and its input can include a text description of the video editing task and a video material 103 to be edited. In some embodiments, the prompt word 112 is used to guide the machine learning model 105 to generate a target file 114 for a target object. The target file 114 may include video-related information. The electronic device 110 can generate a target video based on the target file 114.

[0029] In environment 100, electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phone, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, media computer, multimedia tablet, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, gaming device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of interface for the user (such as "wearable" circuit, etc.). Visual generation model 105 can be implemented, for example, in various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0030] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0031] Figure 1B 1 shows a schematic diagram of an example environment 102 in which embodiments of the present disclosure can be implemented. Figure 1B In the environment 102, it is desirable to train and use such a machine learning model (i.e., machine learning model 130), which is configured for a variety of application environments. For example, the machine learning model 130 can be configured to perform a video editing task, and its input can include prompt words for the video editing task and the video material to be edited.

[0032] like Figure 1BAs shown, the environment 102 includes a model training system 150, a model testing system 160, and a model application system 170. In some embodiments, the trained machine learning model 130 can be directly provided to the model application system 170 for use, or provided to the model testing system 160 for testing and to the model application system 170 for use at the same time.

[0033] Before training, the parameter values ​​of the machine learning model 130 may have initial values, or may have pre-trained parameter values ​​obtained through a pre-training process. The machine learning model 130 may be trained via forward propagation and back propagation, and the parameter values ​​of the machine learning model 130 may be updated and adjusted during the training process. The machine learning model 130' may be obtained after the training is completed. At this point, the parameter values ​​of the machine learning model 130' have been updated, and based on the updated parameter values, the machine learning model 130 may be used to implement the recommendation task in the model application stage.

[0034] In the model training phase, the model training system 150 can train the machine learning model 130 based on a training data set 152 including a plurality of training samples 154. The training samples 154 including sample inputs 155 and true value outputs 156 can be used to train the machine learning model 130. Specifically, the training process can be iteratively performed using a large number of training samples. After the training is completed, the machine learning model 130 may include knowledge about the task to be processed.

[0035] During the model testing phase, the model testing system 160 may use a test data set 162 to test the trained machine learning model 130'. The test data set 162 is similar to the training data set 152 and also includes a plurality of test samples 164, wherein the test sample 164 may include a sample input 165 and a true value output 166. The model testing system 160 uses the test data set 162 to evaluate the various performance indicators of the trained machine learning model 130' to determine whether the model performance of the trained machine learning model 130' meets expectations.

[0036] In the model application stage, the trained machine learning model 130' (at this time, the machine learning model 130' has trained parameter values) can be used to perform the corresponding task. The model application system 170 can use the trained machine learning model 130' to determine the model output 144 based on the model input 172. In the scenario of the video editing task, the model input includes a text description of the video editing task and a plurality of video clips to be edited. The model output can include a clipped target file for generating a clipped video.

[0037] exist Figure 1BIn the figure, the model training system 150, the model testing system 160 and / or the model application system 170 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device may involve any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Servers include but are not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like. Although shown as separate systems in the figure, the model training system 150, the model testing system 160 and / or the model application system 170 can be integrated into a single device / system, and the embodiments of the present disclosure are not limited in this respect.

[0038] Currently, video clips are widely used in different fields, such as video advertising. Generally speaking, an excellent video mix usually contains a logically ordered sequence of clips, attractive spoken text (also known as text description), a strong correlation between the spoken text and the corresponding video clips, appropriate subtitles, etc. However, a lot of effort is required to manually design video creatives, including editing and sequencing, writing spoken text, and ensuring its logical coherence. This complexity often poses a major challenge to users.

[0039] In order to solve at least some of the above problems, in an embodiment of the present disclosure, a scheme for video generation is proposed. In this scheme, multiple video clips for a target object and object information of the target object are obtained, and the multiple video clips respectively have corresponding first identification information. Using a trained machine learning model, a target file is generated based on the multiple video clips and prompt words, the prompt words at least include object information and first identification information of each of the multiple video clips, and the target file at least includes the first identification information corresponding to at least one video clip arranged in sequence among the multiple video clips, a text description of the at least one video clip, and a segmentation result of the text description. Based at least on the target file, a target video for the target object is generated.

[0040] According to the scheme disclosed in the present invention, a trained machine learning model is used to generate a target file including identification information corresponding to the video clips, corresponding text descriptions and segmentation results of the text descriptions based on multiple video clips and prompt words, and a target video for a target object is generated at least based on the target file, which can improve the efficiency of video editing and enhance the quality of the finished film.

[0041] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0042] Figure 2FIG. 2 is a schematic diagram of an example architecture 200 for video generation in the inference phase according to some embodiments of the present disclosure. Figure 1A These embodiments are described with reference to the environment 100 of FIG. These embodiments may be implemented in Figure 1A electronic device 110 or Figure 1B The model is applied in system 170.

[0043] like Figure 2 As shown, in the architecture 200, the electronic device 110 obtains multiple video clips 210 for the target object and object information 220 of the target object. In some embodiments, the type and number of target objects can be set according to different video application fields or application scenarios. For example, in the field of video advertising, the target object can be a certain product (or commodity), or multiple different products of a certain type of product, or multiple types of products, etc., which are not limited here. Assuming that the target object is a certain mobile phone, the multiple video clips 210 can be, for example, multiple different introduction videos for the mobile phone, and the object information 220 can be, for example, the name of the mobile phone or advertising text, etc., which are not limited here.

[0044] In some embodiments, when acquiring multiple video clips 210, the electronic device 110 may determine to select multiple video clips 210 that match the object information 220 from the video material library 230. As an example, the video material library 230 may include one or more video clips corresponding to multiple objects. When acquiring multiple video clips 210 for a target object, the video material library 230 may be queried for multiple video clips for the target object, and then the queried multiple video clips 210 may be acquired.

[0045] It should be noted that the obtained multiple video clips 210 for the target object may be out of order. In addition, the content in the video material library 230 may be updated, for example, new video clips may be added, old video clips that do not meet the requirements may be deleted, etc., and video clips corresponding to new objects may be added, and video clips corresponding to original objects may be deleted.

[0046] Further, the electronic device 110 generates a target file 114 based on the plurality of video clips 210 and the prompt words 112 using the trained machine learning model 105. As an example, the trained machine learning model 105 can be constructed based on a language model (LM). The machine learning model 105 can be configured to perform a video editing task based on a multimodal input. The video editing task can include a video selection task, a text description generation task of a text description corresponding to the video (i.e., a spoken copy), and a text description segmentation task for subtitles.

[0047] In some embodiments, the data format of the target file 114 can be any suitable type of data format, such as JSON data format or other data formats. The JSON protocol can include information for the video track 252, the audio track 254, and the subtitle track 256.

[0048] Here, the plurality of video segments 210 respectively have corresponding first identification information. The prompt word 112 includes at least object information and first identification information of each of the plurality of video segments 210. As an example, the first identification information of each video segment may include an ID number of the video segment. As another example, the first identification information of each video segment may include resource path information of the video segment, such as web page path information, etc. It should be understood that the first identification information of the video segment may be any appropriate information for identifying the video segment or being able to locate the position of the video segment.

[0049] In some embodiments, the prompt word 112 may also include a number requirement of video segments to be selected. According to the number requirement, the machine learning model 105 may be used to select the video segments of the number from the plurality of video segments 210 for video editing.

[0050] Alternatively or additionally, in some embodiments, the prompt word 112 may also include a character count requirement for the text description to be generated. In order to configure audio with a duration matching the selected video segment, the character count of the text description may be limited so that the audio converted from the text description can be aligned with the selected video segment in duration.

[0051] Alternatively or additionally, in some embodiments, the prompt word 112 may also include a segmentation requirement for the text description. The segmentation requirement may correspond to a display requirement for subtitles. In some embodiments, the segmentation requirement for the text description may at least indicate the character number range of the text segmentation unit corresponding to each video frame in the video clip. That is, when displaying subtitles, the text content of a text segmentation unit may be displayed on the screen of each video frame, and the number of characters of the text content may be within a preset range. In addition, the segmentation requirement may also indicate how many text segmentation units each sentence in the text description may be segmented into, and may also indicate which words cannot be cut off, i.e., must be in the same text segmentation unit, and so on.

[0052] In addition, the prompt may also include a description of the video editing requirements. Such a description may be, for example, "You are an advertising video creator, and now you want to create an advertising video that attracts viewers in the following way: naturally connect some video clips, add advertising explanations, and segment the oral explanations to obtain sentence results to make subtitles", etc.

[0053] Alternatively or additionally, in some embodiments, the prompt word 112 may also include example generated content for determining the format of generated content in the target file 114. By providing the example generated content to the machine learning model 105 for reference, the format of the content generated by the machine learning model 105 may be the same as the format of the example generated content. This can facilitate the understanding of the content in the target file 114.

[0054] The target file 114 includes at least first identification information corresponding to at least one video segment arranged in sequence among the plurality of video segments 210, a text description of the at least one video segment, and a segmentation result of the text description. Further, the electronic device 110 generates a target video for the target object based at least on the target file 114. In this way, at least one video segment in the generated target video may be at least one video segment arranged in sequence selected from the plurality of unordered video segments 210. It should be understood that, in some cases, only one video segment may be selected from the plurality of video segments 210 for editing.

[0055] In some embodiments, when generating a target video, the electronic device 110 may obtain at least one video segment based on the first identification information corresponding to the at least one video segment in the target file 114. Then, based on the text-to-speech processing, the electronic device 110 may convert the text description into audio. As an example, the electronic device 110 may convert the text description into audio with the help of a text-to-speech tool. Then, the electronic device 110 may match the at least one video segment arranged in sequence with the audio and the segmentation result of the text description, respectively, to generate the target video.

[0056] In some embodiments, the target file 114 also includes the second identification information of the audio corresponding to the text description. As an example, the second identification information of the audio can include the ID number of the audio. As another example, the second identification information of the audio can include the resource path information of the audio. It should be understood that the second identification information of the audio can be any suitable information that can identify the audio.

[0057] In some embodiments, when generating a target video, based on the second identification information, the electronic device 110 can obtain audio corresponding to the text description, and then can match at least one video segment arranged in sequence with the audio and the segmentation results of the text description to generate the target video.

[0058] In some embodiments, when matching at least one video segment with the audio and the segmentation result of the text description, the electronic device 110 can at least determine the first duration of the at least one video segment and determine the second duration of the audio based on the number of characters in the text description. Then, the electronic device 110 can adjust the number of characters in the text description and / or adjust the second duration of the audio to match the first duration.

[0059] Specifically, the number of characters in the text description to be adjusted and / or the second duration of the audio to be adjusted can be determined based on the difference between the second duration and the first duration. For example, if the duration of the audio is much longer than the duration of at least one video clip, the text description can be streamlined to reduce the number of characters, thereby making the converted audio duration shorter. If the second duration is only slightly longer than the first duration, the speech rate of the audio can be increased to match the first duration. The reverse is also similar and will not be described in detail here. This helps to improve the matching efficiency of the audio duration to the duration of at least one video clip.

[0060] In some embodiments, when matching at least one video segment with the audio and the segmentation results of the text description, the electronic device 110 may also segment the audio into at least one audio segment based on the segmentation results of the text description, and align at least one video segment with at least one audio segment and the segmentation results of the text description in time to generate a target video. In such an embodiment, the video segments arranged in sequence may be aligned in time with the corresponding audio segments and the corresponding one or more text segmentation units, so that at least one video segment is matched with the audio and the segmentation results of the text description. This ensures the accuracy of the matching, thereby improving the quality of the generated target video.

[0061] Through the embodiments of the present disclosure, a trained machine learning model is used to generate a target file including identification information corresponding to the video clips, corresponding text descriptions, and segmentation results of the text descriptions based on multiple video clips and prompt words, and a target video for a target object is generated at least based on the target file, which can improve the efficiency of video editing and enhance the quality of the finished film.

[0062] Figure 3 FIG. 3 is a flowchart showing a process 300 for training a machine learning model 105 according to some embodiments of the present disclosure. The process 300 may be implemented in Figure 1A electronic device 110 or Figure 1B The model is applied in the system 150.

[0063] like Figure 3As shown, in box 310, the electronic device 110 obtains a training data set, and the training data set includes at least a first training sample and first annotation information corresponding to the first training sample. The first training sample includes first object information for a first sample object and multiple video clips. The first annotation information includes a true value video clip sequence for the first sample object, a corresponding first true value text description, and a segmentation result of the first true value text description.

[0064] The annotation information discussed herein may be the true value information corresponding to the training sample, and the true value information may be included in the training data set. Such a training data set may be used for the machine learning model 105 to be trained to perform the video editing task. The video editing task may include a video selection task, a text description generation task of the text description corresponding to the video (i.e., the spoken copy), and a text description segmentation task for subtitles.

[0065] The training samples discussed herein may include a large number of sample objects. In some embodiments, the type and number of individual sample objects in the sample objects may be set according to different video application fields or application scenarios. For example, in the field of video advertising, a single sample object may be a certain product (or commodity), or may be a plurality of different products of a certain type of product, or may be a plurality of types of products, and so on, without limitation here. Assuming that a single sample object is a certain mobile phone, the corresponding plurality of video clips may be, for example, a plurality of different introduction videos for the mobile phone, and the corresponding sample object information may be, for example, the name of the mobile phone or an advertisement text, and so on, without limitation here.

[0066] It should be noted that the multiple video clips in the first training sample may be multiple unordered video clips. The video clips in the true value video clip sequence may be ordered video clips, that is, video clips arranged in sequence. The machine learning model 105 may be configured to perform a video editing task based on multimodal input.

[0067] In block 320, further, the electronic device 110 performs a first training task on the model based on the training data set in the following manner: The first training task may be referred to as a composite training task.

[0068] Specifically, the electronic device 110 uses the machine learning model 105 to be trained to generate a prediction result for the first sample object based on the first training sample, and the prediction result includes a first predicted video clip sequence, a corresponding first predicted text description, and a segmentation result of the first predicted text description.

[0069] When the machine learning model 105 to be trained is used to generate a first predicted video segment sequence, a corresponding first predicted text description, and a segmentation result of the first predicted text description based on the first training sample, a training intermediate file can be generated based on multiple video segments and training prompt words in the first training sample. The training prompt words can at least include object information of the first sample object and identification information of each of the multiple video segments. The training intermediate file can include identification information of the first predicted video segment sequence, as well as the first predicted text description and the segmentation result of the first predicted text description. Then, the first predicted video segment sequence can be obtained based on the identification information of the first predicted video segment sequence.

[0070] Further, the electronic device 110 trains the machine learning model 105 based on the difference between the prediction result and the first annotation information. The training goal of the machine learning model 105 to be trained is to make the prediction result output by the machine learning model 105 as close as possible to the first annotation information. Therefore, during training, a corresponding training loss can be defined based on the difference between the prediction result and the first annotation information. The training loss can be expressed as any appropriate loss function that can determine the difference between the prediction result and the first annotation information. The difference between the prediction result and the first annotation information can be minimized or reduced to meet the predetermined goal. Then, the trained machine learning model 105 can be used Figure 4 in the testing phase.

[0071] In some embodiments, the electronic device 110 may perform pre-training on the model by executing a second training task on the machine learning model 105. The second training task may be referred to as a video mixing task for a video track.

[0072] The execution of the second training task may include: the electronic device 110 may select a second training sample and a second annotation information from the training data set. The second training sample may include the second object information of the second sample object and a plurality of unordered video clips. The plurality of video clips may include at least one video clip matching the second object information and at least one video clip unrelated to the second object information. The second annotation information may include an ordered sequence of video clips matching the second object information. Using the machine learning model 105 to be trained, a second predicted video clip sequence may be generated based on the second training sample. Based on the difference between the second predicted video clip sequence and the ordered video clip sequence, the machine learning model 105 may be trained.

[0073] In the pre-training stage of this video remix task, the training goal of the machine learning model 105 to be trained is to make the second predicted video segment sequence predicted by the machine learning model 105 as close as possible to the true ordered video segment sequence. Therefore, in this pre-training stage, the corresponding training loss can be defined based on the difference between the second predicted video segment sequence and the ordered video segment sequence. The training loss can be expressed as any appropriate loss function that can judge the difference between the second predicted video segment sequence and the ordered video segment sequence. The difference between the second predicted video segment sequence and the ordered video segment sequence can be minimized or reduced to meet the corresponding predetermined goal.

[0074] In some embodiments, the electronic device 110 may perform pre-training on the model by performing a third training task on the model. The third training task may be referred to as a text prediction task for an audio track.

[0075] The execution of the third training task may include: the electronic device 110 may select a third training sample and third annotation information from the training data set. The third training sample may include third object information of the third sample object and a video clip sequence matching the third object information. The third annotation information may include a second true value text description corresponding to the video clip sequence. Using the machine learning model 105 to be trained, a second predicted text description may be generated based on the third training sample. Based on the difference between the second predicted text description and the second true value text description, the machine learning model 105 may be trained.

[0076] In the pre-training stage of this text prediction task, the training goal of the machine learning model 105 to be trained is to make the second predicted text description predicted by the machine learning model 105 as close as possible to the second true text description. Therefore, in this pre-training stage, the corresponding training loss can be defined based on the difference between the second predicted text description and the second true text description. The training loss can be expressed as any appropriate loss function that can judge the difference between the second predicted text description and the second true text description. The difference between the second predicted text description and the second true text description can be minimized or reduced to meet the corresponding predetermined goal.

[0077] In some embodiments, the electronic device 110 may perform training on the model by performing a fourth training task on the model. The fourth training task may be referred to as a text segmentation task for a subtitle track.

[0078] The execution of the fourth training task may include: the electronic device 110 may select a fourth training sample and fourth annotation information from the training data set. The fourth training sample may include fourth object information of a fourth sample object and a video clip sequence matching the fourth object information. The fourth annotation information may include a segmentation result of a third true value text description corresponding to the video clip sequence. Using the machine learning model 105 to be trained, a segmentation result of a third predicted text description may be generated based on the fourth training sample. Based on the difference between the segmentation result of the third predicted text description and the segmentation result of the third true value text description, the machine learning model 105 may be trained.

[0079] In the training phase of this text segmentation task, the training goal of the machine learning model 105 to be trained is to make the segmentation result of the third predicted text description predicted by the machine learning model 105 as close as possible to the segmentation result of the third true text description. Therefore, in this pre-training phase, the corresponding training loss can be defined based on the difference between the segmentation result of the third predicted text description and the segmentation result of the third true text description. The training loss can be expressed as any appropriate loss function that can judge the difference between the segmentation result of the third predicted text description and the segmentation result of the third true text description. The difference between the segmentation result of the third predicted text description and the segmentation result of the third true text description can be minimized or reduced to meet the corresponding predetermined goal.

[0080] In some embodiments, in the first training sample, the information corresponding to each video segment may include identification information of the video segment. The identification information of the video segment may include, for example, an ID number of each video segment, or may include a resource path such as a webpage address of each video segment. It should be understood that the identification information of the video segment may be any appropriate information for identifying the video segment or being able to locate the position of the video segment.

[0081] Alternatively or additionally, in some embodiments, the information corresponding to each video segment may further include a start symbol indicating the beginning of the information corresponding to the video segment, and an end symbol indicating the end of the information corresponding to the video segment. The information corresponding to each video segment may be between the start symbol and the end symbol. The start symbol and the end symbol may be represented by appropriate characters, labels or other symbols, etc., which are not limited here. Through the start symbol and the end symbol of each video segment, each video segment can be quickly identified.

[0082] Alternatively or additionally, in some embodiments, the information corresponding to each video segment may also include at least one video frame extracted from the video segment and a rewriting result of the text description corresponding to the video segment. The rewriting result may be, for example, a rewriting result of the text description corresponding to the extracted video frame. The rewriting result may be used in the training phase of the machine learning model 105, which is conducive to improving the accuracy of the model prediction and enhancing the robustness of the model.

[0083] Alternatively or additionally, in some embodiments, the information corresponding to each video segment may further include a target number of characters that matches the video segment. The target number of characters may be a target number of characters for a text description of the audio corresponding to the video segment. By setting the target number of characters that matches the video segment, the duration of the video segment may be matched with the duration of the corresponding audio.

[0084] The following will discuss in detail the construction method of the training data set. The training data set of the model in the training phase and the test data set in the test phase can both be included in the data set of the model. It should be understood that the construction of the data set of the model can uniformly adopt the construction method discussed below. In other embodiments, the construction of the test data set of the model can also adopt other appropriate construction methods separately.

[0085] In some embodiments, the electronic device 110 may construct a training data set in the following manner. The electronic device 110 may divide the video material into a group of video clips to obtain multiple groups of video clips. Then, based on the selection requirements for the multiple groups of video clips, the electronic device 110 may select at least one group of video clips from the multiple groups of video clips, and construct the training data set based on the at least one group of video clips.

[0086] When constructing a training dataset, the video material may include a large amount of video material about multiple sample objects. Each video material may be split into a series of video clips. For example, in the field of video advertising, the video material may be an advertising video for multiple products.

[0087] In some embodiments, when segmenting the video material into a group of video segments, if the video material includes a text description, the electronic device 110 can use a speech recognition model to extract the corresponding first text description and the timestamps of multiple punctuation marks in the first text description from the video material. Then, based on the timestamps of the multiple punctuation marks, the electronic device 110 can segment the video material into a group of video segments. The speech recognition model can be, for example, an ASR (automatic speech recognition) model.

[0088] In some embodiments, if the video material does not have a text description, based on the degree of change of the visual features of each video frame in the video material, the electronic device 110 can segment the video material into a group of video segments. As an example, the visual feature difference between the continuous frames and the split frames of the video frames of the video material can be measured (the difference can be set to be greater than a preset threshold), so as to segment the video material into a group of video segments.

[0089] In some embodiments, the selection requirement may include a requirement for the number of video segments in each of the multiple groups of video segments. Alternatively or additionally, in some embodiments, the selection requirement may also include a requirement for the text fluency of the text description extracted from the video material. Alternatively or additionally, in some embodiments, the selection requirement may also include a requirement for the relevance between the text description and the object information.

[0090] Such selection requirements are intended to retain high-quality video materials, which can then be optimized and segmented through manual annotation or tool annotation. For example, screening criteria may include: the video length should not exceed 120 seconds or other duration; the number of video clips should be between 2 and 8, or within other set ranges; the text description should be fluent; the text description should be related to the video and object information. In this way, high-quality training data sets can be screened out.

[0091] Based on such a training dataset, a machine learning model is used to perform the video track remixing task, which can correctly select and sort the given unordered clips. A machine learning model is used to perform the script prediction task of the audio track, aiming to generate attractive and logically relevant spoken text for each video clip. A machine learning model can also be used to perform the script segmentation task of the subtitle track, including predicting appropriate punctuation for the spoken text. The output of the composite training task of all three tracks can be post-processed into JSON protocol or other suitable protocols. JSON protocol or other suitable protocols can be converted into video creatives, so that the quality can be systematically evaluated.

[0092] Figure 4 FIG. 4 is a flowchart of a process 400 for evaluating a machine learning model during a testing phase according to some embodiments of the present disclosure. The process 400 may be implemented in Figure 1A electronic device 110 or Figure 1B In the model testing system 160.

[0093] In block 410 , the electronic device 110 acquires a test data set, where the test data set includes a test sample, and the test sample includes object information for a test object and a plurality of video clips.

[0094] In some embodiments, the type and number of test objects can be set according to different video application fields or application scenarios. For example, in the field of video advertising, the test object can be a certain product (or commodity), or multiple different products of a certain type of product, or multiple types of products, etc., without limitation here. Assuming that the test object is a certain mobile phone, the multiple video clips can be, for example, multiple different introduction videos for the mobile phone, and the object information can be, for example, the name of the mobile phone or the advertising text, etc., without limitation here.

[0095] In block 420, the electronic device 110 further generates a predicted video segment sequence and its corresponding text description and a segmentation result of the text description based on the test sample using the trained machine learning model 105. The video segment sequence includes at least two video segments arranged in sequence from the plurality of video segments.

[0096] When using the trained machine learning model 105 to generate a predicted video clip sequence and its corresponding text description and segmentation results of the text description based on the test sample, a test file can be generated based on multiple video clips and test prompt words. The test prompt words can at least include object information of the test object and identification information of each of the multiple video clips. The test file can include identification information of the predicted video clip sequence, as well as the predicted text description and segmentation results of the text description. Then, the predicted video clip sequence can be obtained based on the identification information of the predicted video clip sequence.

[0097] In block 430, the electronic device 110 further determines the metric values ​​of the performance indicators corresponding to the predicted video clip sequence, the text description, and the segmentation results of the text description. How to determine the metric values ​​of the performance indicators corresponding to the predicted video clip sequence, the text description, and the segmentation results of the text description will be discussed in detail below.

[0098] In some embodiments, when determining the metric values ​​of the performance indicators corresponding to the predicted video clip sequence, text description, and segmentation results of the text description, the electronic device 110 may utilize another machine learning model to generate metric values ​​of the performance indicators corresponding to the predicted video clip sequence, text description, and segmentation results of at least one of the text description based on the scoring requirements for the performance indicators corresponding to each of the video clip sequence, text description, and segmentation results of the text description.

[0099] In such an embodiment, another machine learning model may be constructed based on a language model. The another machine learning model may be configured to receive a prompt word for model evaluation, and generate corresponding multiple metrics based on the scoring requirements for multiple performance indicators in the prompt word. It should be understood that when the predicted video clip sequence, text description, and segmentation result of the text description are evaluated separately, the another machine learning model may be used for evaluation respectively, or other methods may be used for evaluation.

[0100] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted video clip sequence, the electronic device 110 may determine the metric values ​​of the corresponding coincidence performance indicator and accuracy performance indicator. It should be understood that in other embodiments, other performance indicators for the predicted video clip sequence may also be used for evaluation.

[0101] In some embodiments, when determining the metric value of the coincidence performance index corresponding to the predicted video segment sequence, the electronic device 110 can determine the metric value of the coincidence performance index based on the intersection of the video segments included between the predicted video segment sequence and the true value video segment sequence corresponding to the test sample. As an example, the larger the intersection of the video segments of the two, the larger the metric value of the coincidence performance index corresponding to the predicted video segment sequence. This means that if the video segments in the predicted video segment sequence are exactly the same as the video segments in the true value video segment sequence, that is, the two completely overlap, the metric value of the coincidence performance index is also the largest.

[0102] In some embodiments, when determining the metric value of the accuracy performance indicator corresponding to the predicted video segment sequence, the electronic device 110 can determine the metric value of the accuracy performance indicator based on whether the predicted video segment sequence is the same as the video segments arranged in order in the true value video segment sequence. As an example, when the video segments of the two are the same, if the arrangement order of the video segments is also the same, it means that the higher the accuracy, the greater the metric value of the accuracy performance indicator. If the video segments of the two are exactly the same and the arrangement order is also exactly the same, the metric value of the accuracy performance indicator can reach the maximum.

[0103] By using the overlap index and the accuracy index to evaluate the predicted video clip sequence, the quality of the subsequent film can be improved.

[0104] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, another machine learning model can be used. As an example, another machine learning model can perform an evaluation based on the input prompt words for the text description evaluation. The prompt words may include evaluation requirements and scoring requirements for each performance indicator. For example, the evaluation requirement may be "You need to evaluate the text description based on the following performance indicators: authenticity index, text fluency index, contextual coherence index, logic index, picture relevance index, object relevance index". It should be understood that any other appropriate performance indicators can also be used to evaluate the text description.

[0105] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the authenticity indicator corresponding to the predicted text description.

[0106] Specifically, it can be determined whether the predicted text description is in fact consistent with the corresponding video clip and object information. In other words, the authenticity index can be used to determine whether the predicted text description is true or fictitious relative to the corresponding video clip and object information. For example, if price information appears in the text description, but the price information cannot be verified from the corresponding object information or video clip, it can be determined that the authenticity of the predicted text description is low, that is, the corresponding authenticity index has a low metric value.

[0107] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the text fluency indicator corresponding to the predicted text description. Text fluency may refer to text coherence.

[0108] Specifically, it is possible to determine whether the text description is fluent, whether there are typos, whether it is easy to read aloud, etc., and set scoring intervals corresponding to different fluency levels to determine the measurement value of the text description with respect to the text fluency index.

[0109] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the context coherence indicator corresponding to the predicted text description.

[0110] Specifically, it can determine whether the text is contradictory, whether it repeats large sections, whether it is not concise enough, whether it is semantically connected, etc., and set scoring ranges corresponding to different degrees of coherence to determine the measurement value of the text description regarding the contextual coherence indicator.

[0111] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the logic indicator corresponding to the predicted text description.

[0112] Specifically, it is possible to determine whether the predicted text description has logical errors, whether it is logically connected, etc., so as to determine the metric value of the text description with respect to the logic index. For example, for the text description "The lens is coated with an oleophobic layer to prevent oil stains", this is logical and can be evaluated with a higher metric value. For another example, for the text description "Tea brewed in summer cools faster than tea brewed in winter", this is unreasonable and can be evaluated with a lower metric value.

[0113] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the correlation indicator between the predicted text description and the video frames in the predicted video clip sequence. In such an embodiment, some video frames need to be extracted from the predicted video clip sequence for calculating the metric value of the picture correlation indicator.

[0114] Specifically, it is possible to determine whether the text description corresponding to the picture content of the video frame is related to the picture content, or whether the two are highly consistent, or whether the two are unrelated to each other, etc., and set different scores for different degrees of relevance, thereby determining the metric value of the correlation index between the predicted text description and the video frame in the predicted video clip sequence.

[0115] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the correlation indicator between the predicted text description and the object information of the test object.

[0116] Specifically, it is possible to determine whether the predicted text description is related to the object information of the test object, or whether the test object is described in detail from multiple aspects, or whether the two are unrelated to each other, etc., and set different scores for different degrees of relevance, thereby determining the metric value of the correlation index between the predicted text description and the object information of the test object.

[0117] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the character length indicator of the text description based on the difference between the target number of characters of the predicted video clip sequence and the number of characters contained in the text description.

[0118] In such an embodiment, the duration of the corresponding audio can be determined based on the duration of the predicted video clip sequence, and then the target number of characters of the predicted video clip sequence can be determined. As an example, if the difference between the target number of characters of the predicted video clip sequence and the number of characters contained in the text description is smaller, or the target number of characters is closer to the number of characters contained in the text description, it can be determined that the metric value of the character length index of the predicted text description is larger. Conversely, if the difference between the target number of characters of the predicted video clip sequence and the number of characters contained in the text description is larger, the metric value of the character length index of the predicted text description can be smaller.

[0119] In the above embodiment, the predicted text description is evaluated based on authenticity index, text fluency index, contextual coherence index, logic index, screen relevance index, object relevance index, and character length index, which is beneficial to improving the quality of the model in the application stage.

[0120] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted segmentation result of the text description, based on whether the segmentation result of the text description meets at least one segmentation requirement, the electronic device 110 may determine the metric value of the performance indicator corresponding to the segmentation result. For ease of understanding, the segmentation requirement may correspond to a display requirement for subtitles on a screen of a video frame.

[0121] In some embodiments, at least one segmentation requirement may indicate a requirement for the number of characters of a text segmentation unit corresponding to each video frame in the video segment sequence. That is, when displaying subtitles, the text content of a text segmentation unit may be displayed on the screen of each video frame. The number of characters of each text segmentation unit may be required to be within a preset character number range.

[0122] Alternatively or additionally, in some embodiments, at least one segmentation requirement may indicate a requirement on the number of text segmentation units into which the text content between specified punctuation marks of the text description is divided. The specified punctuation marks may include, for example, a period, a comma, or other punctuation marks, etc., which are not limited here. For example, each sentence in the text description may be required to be segmented into several text segmentation units according to the period.

[0123] Alternatively or additionally, in some embodiments, at least one segmentation requirement may indicate a segmentation requirement for partial words in the text description. For example, it may be required that the partial words cannot be cut off, that is, must be in the same text segmentation unit. As an example, such partial words may be commonly used basic vocabulary.

[0124] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted segmentation result of the text description, the corresponding metric value can be determined for each of the above segmentation requirements. In some embodiments, when the predicted text description meets multiple segmentation requirements at the same time, it can be determined that its metric value is larger. Correspondingly, the fewer segmentation requirements are met, the smaller the metric value can be.

[0125] By evaluating the predicted segmentation results of the text description with respect to the above-mentioned multiple segmentation requirements, the readability of the subtitles in the subsequent film can be improved.

[0126] In block 440, further, the electronic device 110 determines the model performance of the trained machine learning model 105 based on the metric value of the performance indicator. Thus, the model performance of the machine learning model 105 can be determined by evaluating the multiple performance indicators discussed above for the predicted video clip sequence, text description, and the segmentation result of the text description.

[0127] It should be understood that if the performance meets expectations, the machine learning model 105 can be put into use for reasoning (or application). If the performance does not meet expectations, it is possible to continue to return to the model training phase and update the machine learning model 105 until the evaluated model performance meets the expected performance.

[0128] The evaluation process of the machine learning model is beneficial to improving the quality of subsequent models in the application stage.

[0129] Figure 5 1 is a flow chart of a method 500 for video generation according to some embodiments of the present disclosure. The method 500 may be implemented in Figure 1A The electronic device 110 is referred to as Figure 1A The method 500 is described with reference to the environment 100. In block 510, a plurality of video clips for a target object and object information of the target object are obtained, and the plurality of video clips respectively have corresponding first identification information.

[0130] In box 520, a target file is generated based on multiple video clips and prompt words using a trained machine learning model, the prompt words at least include object information and first identification information of each of the multiple video clips, and the target file at least includes the first identification information corresponding to at least one video clip arranged in sequence among the multiple video clips, a text description of the at least one video clip, and a segmentation result of the text description.

[0131] In block 530 , a target video for a target object is generated based at least on the target file.

[0132] In some embodiments, generating a target video includes: obtaining at least one video segment based on first identification information corresponding to at least one video segment in a target file; converting a text description into audio based on text-to-speech processing; and matching at least one video segment arranged in sequence with the audio and the segmentation results of the text description, respectively, to generate a target video.

[0133] In some embodiments, the target file also includes second identification information of audio corresponding to the text description, and generating the target video includes: based on the second identification information, obtaining the audio corresponding to the text description; and matching at least one video segment arranged in sequence with the audio and the segmentation results of the text description, respectively, to generate the target video.

[0134] In some embodiments, matching at least one video segment with the audio and the segmentation results of the text description respectively includes at least: determining a first duration of at least one video segment; determining a second duration of the audio based on the number of characters in the text description; and matching the second duration with the first duration by adjusting the number of characters in the text description and / or the second duration.

[0135] In some embodiments, matching at least one video segment with the audio and the segmentation results of the text description respectively also includes: segmenting the audio into at least one audio segment based on the segmentation results of the text description; and temporally aligning at least one video segment with the at least one audio segment and the segmentation results of the text description respectively to generate a target video.

[0136] In some embodiments, the prompt word also includes at least one of the following: a requirement on the number of video clips to be selected, a requirement on the number of characters of the text description to be generated, example generated content for determining the format of the generated content in the target file, and a requirement on the segmentation of the text description.

[0137] In some embodiments, the segmentation requirement for the text description at least indicates a range of the number of characters of the text segmentation unit corresponding to each video frame in the video segment.

[0138] In some embodiments, obtaining a plurality of video clips includes: determining to select a plurality of video clips matching the object information from a video material library.

[0139] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 6 An exemplary structural block diagram of an apparatus 600 for video generation according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0140] like Figure 6 As shown, the device 600 includes an acquisition module 610, which is configured to acquire multiple video clips for a target object and object information of the target object, and the multiple video clips respectively have corresponding first identification information; a file generation module 620, which is configured to generate a target file based on the multiple video clips and prompt words using a trained machine learning model, the prompt words at least include object information and the first identification information of each of the multiple video clips, and the target file at least includes the first identification information corresponding to at least one video clip arranged in sequence among the multiple video clips, a text description of the at least one video clip, and a segmentation result of the text description; and a video generation module 630, which is configured to generate a target video for the target object based on at least the target file.

[0141] In some embodiments, the video generation module 630 is further configured to obtain at least one video segment based on first identification information corresponding to at least one video segment in the target file; convert the text description into audio based on text-to-speech processing; and match at least one video segment arranged in sequence with the audio and the segmentation results of the text description, respectively, to generate a target video.

[0142] In some embodiments, the target file also includes second identification information of audio corresponding to the text description, and the video generation module 630 is further configured to obtain the audio corresponding to the text description based on the second identification information; and match at least one video segment arranged in sequence with the audio and the segmentation results of the text description to generate a target video.

[0143] In some embodiments, device 600 is also configured to determine a first duration of at least one video segment; determine a second duration of the audio based on the number of characters in the text description; and match the second duration with the first duration by adjusting the number of characters in the text description and / or the second duration.

[0144] In some embodiments, the device 600 is also configured to segment the audio into at least one audio segment based on the segmentation result of the text description; and temporally align at least one video segment with the at least one audio segment and the segmentation result of the text description to generate a target video.

[0145] In some embodiments, the prompt word also includes at least one of the following: a requirement on the number of video clips to be selected, a requirement on the number of characters of the text description to be generated, example generated content for determining the format of the generated content in the target file, and a requirement on the segmentation of the text description.

[0146] In some embodiments, the segmentation requirement for the text description at least indicates a range of the number of characters of the text segmentation unit corresponding to each video frame in the video segment.

[0147] In some embodiments, the acquisition module 610 is further configured to determine to select a plurality of video segments matching the object information from the video material library.

[0148] The units and / or modules included in the device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 600 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0149] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, Figure 1A The computing device 110 in FIG.

[0150] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to implement Figure 1A Electronic device 110 or Figure 6 Device 600.

[0151] like Figure 7 As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0152] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 700.

[0153] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0154] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0155] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0156] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0157] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0158] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0159] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0160] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as an update, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0161] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for video generation, comprising: Acquire a plurality of video clips for a target object and object information of the target object, wherein the plurality of video clips respectively have corresponding first identification information; Generate a target file based on the multiple video clips and prompt words by using the trained machine learning model, wherein the prompt words at least include the object information and the first identification information of each of the multiple video clips, and the target file at least includes the first identification information corresponding to at least one video clip arranged in sequence among the multiple video clips, a text description of the at least one video clip, and a segmentation result of the text description; as well as A target video for the target object is generated based at least on the target file.

2. The method according to claim 1, wherein generating the target video comprises: Acquire the at least one video segment based on the first identification information corresponding to the at least one video segment in the target file; Based on text-to-speech processing, convert the text description into audio; and The at least one video segment arranged in sequence is matched with the audio and the segmentation result of the text description respectively to generate the target video.

3. The method according to claim 1, wherein the target file further includes second identification information of the audio corresponding to the text description, and The generating of the target video comprises: Based on the second identification information, acquiring audio corresponding to the text description; as well as The at least one video segment arranged in sequence is matched with the audio and the segmentation result of the text description respectively to generate the target video.

4. The method according to claim 2 or 3, wherein matching the at least one video segment with the audio and the segmentation results of the text description respectively comprises at least: determining a first duration of the at least one video segment; Determining a second duration of the audio based on the number of characters in the text description; as well as The second duration is matched with the first duration by adjusting the number of characters in the text description and / or the second duration.

5. The method according to claim 2 or 3, wherein matching the at least one video segment with the audio and the segmentation results of the text description respectively further comprises: Based on the segmentation result of the text description, segment the audio into at least one audio segment; as well as The at least one video segment is respectively aligned in time with the at least one audio segment and the segmentation result of the text description to generate the target video.

6. The method according to claim 1, wherein the prompt word further comprises at least one of the following: The number of video clips to be selected is required. The number of characters required for the text description to be generated. Example generated content for determining the format of the generated content in the target file, Requirements for segmentation of text descriptions.

7. The method according to claim 5, wherein the segmentation requirement for the text description at least indicates a range of the number of characters of a text segmentation unit corresponding to each video frame in a video segment.

8. The method according to claim 1, wherein obtaining the plurality of video clips comprises: Determine to select the multiple video clips matching the object information from a video material library.

9. A device for video generation, comprising: An acquisition module is configured to acquire a plurality of video clips for a target object and object information of the target object, wherein the plurality of video clips respectively have corresponding first identification information; A file generation module is configured to generate a target file based on the multiple video clips and prompt words using a trained machine learning model, wherein the prompt words at least include the object information and the first identification information of each of the multiple video clips, and the target file at least includes the first identification information corresponding to at least one video clip arranged in sequence among the multiple video clips, a text description of the at least one video clip, and a segmentation result of the text description; as well as The video generation module is configured to generate a target video for the target object based at least on the target file.

10. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images