Role view angle video generation method, related device and medium

Abstract

The invention provides a role viewing angle video generation method, a related device and a medium. The method comprises the following steps: acquiring a source video with a target role and source background description information of the source video; extracting source script information from the source video; obtaining guide language template information corresponding to the role script conversion model; on the basis of the source script information, the source background description information and the guide language template information, role script information of a target role view angle is generated by utilizing a role script conversion model; and generating diffusion guide information based on the role script information, the picture information corresponding to the target role and the source background description information, and generating a role view angle video of a target role view angle based on the diffusion guide information. According to the invention, the accuracy of generating auxiliary videos of different figure view angles for film and television works can be improved. The method can be applied to various scenes such as big data, artificial intelligence and cloud technology.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a method for generating a character perspective video, a related device and a medium. Background Art

[0002] At present, in some business scenarios of showing narrative movies or TV series, the film and television works often need to be shown to each object from a single perspective. This display method often limits the object from experiencing the film and television works from different perspectives, such as the psychological activities of the characters and the picture information from the perspective of the characters. This information is sometimes critical to fully understand the content of the film and television works. This key information cannot be transmitted to the object, resulting in low efficiency of information transmission.

[0003] Therefore, there is an urgent need to generate auxiliary videos from different perspectives for film and television works to improve the efficiency of information transmission. The related technology is a technology that generates short videos from different perspectives for characters in a short video based on the description information of a short video, but this technology is limited to short videos, and it is generated based on the description information of the short video, which cannot be accurately obtained, resulting in the low accuracy of the auxiliary short videos generated from different perspectives of the characters in the technology. How to generate auxiliary videos from different perspectives of characters for film and television works with high accuracy is still a problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, a related device and a medium for generating a character perspective video, which can improve the accuracy of generating auxiliary videos from different character perspectives for film and television works.

[0005] According to one aspect of the present disclosure, a method for generating a character perspective video is provided, the method comprising: Acquire a source video having a target character and source background description information of the source video; Extracting source script information from the source video; Obtain the introductory text template information corresponding to the role script conversion model; Based on the source script information, the source background description information and the guide template information, the role script information from the perspective of the target character is generated by using the role script conversion model; Based on the role script information, the picture information corresponding to the target role, and the source background description information, diffusion guidance information is generated, and based on the diffusion guidance information, a role perspective video from the perspective of the target role is generated.

[0006] According to one aspect of the present disclosure, a device for generating a character perspective video is provided, the device comprising: A first acquisition unit, configured to acquire a source video having a target character and source background description information of the source video; An extraction unit, used to extract source script information from the source video; The second acquisition unit is used to acquire the introductory language template information corresponding to the role script conversion model; A first generating unit, configured to generate role script information from the perspective of the target role using the role script conversion model based on the source script information, the source background description information and the guide template information; The second generating unit is used to generate diffusion guidance information based on the role script information, the picture information corresponding to the target role, and the source background description information, and generate a role perspective video from the perspective of the target role based on the diffusion guidance information.

[0007] Optionally, the first generating unit includes: An acquisition module, used to acquire image information corresponding to the target character; The first generation module is used to generate introductory words based on the source script information, the picture information, and the source background description information using the introductory words template information, and input the introductory words into the role script conversion model to obtain the role script information.

[0008] Optionally, the source script information includes source line information and source scene information, the role script information includes role script line information and role script scene information, the introductory language template information includes first introductory language template information and second introductory language template information, and the introductory language includes first introductory language and second introductory language; The first generating module is used for: Based on the source dialogue information, the source scene information, the picture information and the source background description information, the first guide language template information is used to generate a first guide language, and the first guide language is input into the role script conversion model to obtain the role script dialogue information; Based on the source scene information and the source background description information, the second guide language is generated using the second guide language template information, and the second guide language is input into the role-script conversion model to obtain the role-script scene information.

[0009] Optionally, the role-script conversion model includes a first sub-model and a second sub-model; The first generating unit is used for: Based on the guide template information, the source script information, and the source background description information, the first sub-model is used to generate an instance to obtain a role script prompt instance; Based on the role script prompt instance, the script is generated through the second sub-model to obtain the role script information.

[0010] Optionally, the second acquiring unit includes: A segmentation module, used for segmenting the source video to obtain a plurality of source video frames arranged in time sequence; A second generation module is used to input the plurality of source video frames into a pre-trained video script description generation model for description generation, so as to obtain video frame script description information of each of the plurality of source video frames; An integration module is used to integrate the video frame script description information of each of the multiple source video frames to obtain the source script information.

[0011] Optionally, the video script description generation model includes a feature extraction network and a script description generation network; The second generation module comprises: An extraction submodule, configured to extract features of each of the source video frames based on the feature extraction network to obtain target video frame features of the source video frame; A generation submodule is used to generate a script based on the target video frame features using the script description generation network to obtain the video frame script description information of the source video frame.

[0012] Optionally, the extraction submodule is used to: For each of the source video frames, performing multi-scale downsampling on the source video frame based on the feature extraction network to obtain a first downsampling feature, a second downsampling feature, and a third downsampling feature, wherein the feature dimensions of the first downsampling feature, the second downsampling feature, and the third downsampling feature are increased in sequence; Performing dimensionality reduction processing on the first down-sampling feature, the second down-sampling feature, and the third down-sampling feature, respectively, to obtain a first dimensionality reduction feature corresponding to the first down-sampling feature, a second dimensionality reduction feature corresponding to the second down-sampling feature, and a third dimensionality reduction feature corresponding to the third down-sampling feature; Performing splicing processing on the first dimensionality reduction feature, the second dimensionality reduction feature, and the third dimensionality reduction feature to obtain an initial video frame feature of the source video frame; The initial video frame features of the source video frame and the initial video frame features of a previous source video frame of the source video frame are spliced ​​to obtain target video frame features of the source video frame.

[0013] Optionally, the integration module is used to: splicing the video frame script description information of each of the multiple source video frames to obtain preliminary script information; The preliminary script information is polished to obtain the source script information.

[0014] Optionally, the second generating unit is used for: Encoding the source background description information to obtain background description encoding features; Encoding the role script information to obtain role script encoding features; Extracting features from the image information to obtain features of the target character image; Based on the background description coding features, the role script coding features, and the target role image features, conditional information is generated through a preset conditional fusion network to obtain the diffusion guidance information.

[0015] Optionally, the second generating unit is used to: guide a diffusion model based on the diffusion guiding information to generate a character perspective video of the target character perspective; The diffusion model is trained in the following way: A sample acquisition unit, used to acquire a sample video having a sample character, sample background description information of the sample video, and an expected character perspective video of the sample character; A sample extraction unit, used to extract sample script information from the sample video; A sample script generating unit, configured to generate sample role script information from the perspective of the sample role based on the sample script information, the sample background description information and the preset introductory language template information and using a role script conversion model; A sample guide information generating unit, configured to generate sample guide information based on the sample role script information, the sample picture information corresponding to the sample role, and the sample background description information; A sample video generating unit, configured to generate a video through a target model based on the sample guiding information to obtain a sample character perspective video of the sample character perspective; The first training unit is used to train the target model based on the sample role perspective video and the expected role perspective video to obtain the diffusion model.

[0016] Optionally, the sample video includes a plurality of sample video frames; the target model includes a diffusion network, a denoising network, and a decoding network; The sample video generating unit is used for: A diffusion module, configured to perform diffusion processing on the sample video frame encoding features of each of the sample video frames based on the diffusion network to obtain a sample latent space feature vector; A denoising module, configured to perform denoising processing on the sample latent space feature vector through the denoising network based on the sample guide information to obtain a sample video frame denoising feature; A decoding module, used for decoding the denoising features of the sample video frames based on the decoding network to obtain predicted video frames of each of the sample video frames; The video generation module is used to obtain the sample character perspective video based on the predicted video frames of each of the multiple sample video frames.

[0017] Optionally, the denoising network includes an upsampling module and a downsampling module; The denoising module is used to: Performing splicing processing on the sample guide information and the sample latent space feature vector to obtain a sample denoising splicing feature; Downsampling the sample denoising splicing features based on the downsampling module to obtain video frame downsampling features; The down-sampled features of the video frame are up-sampled based on the up-sampling module to obtain the denoising features of the sample video frame.

[0018] Optionally, the denoising module is used to: Performing splicing processing on the sample guide information and the sample latent space feature vector to obtain a sample denoising splicing feature; Based on the sample denoising splicing features, performing a first denoising process through the denoising network to obtain a first denoising feature; Based on the sample latent space feature vector, performing a second denoising process through the denoising network to obtain a second denoising feature; The first denoising feature and the second denoising feature are weighted to obtain the sample video frame denoising feature.

[0019] Optionally, the sample video includes a plurality of sample video frames; The first training unit is used for: Determining a first loss sub-function based on the sample character perspective video and the expected character perspective video; Determining a second loss sub-function based on the sample video frame denoising features corresponding to each of the plurality of sample video frames and a preset benchmark denoising feature; Based on the first loss sub-function and the second loss sub-function, obtaining a first loss function; Based on the first loss function, the target model is trained to obtain the diffusion model.

[0020] Optionally, determining the second loss sub-function based on the sample video frame denoising features corresponding to each of the plurality of sample video frames and a preset reference denoising feature includes: For each of the sample video frames, a regularization term is calculated based on the sample video frame denoising feature and the reference denoising feature to obtain a regularization term calculation result; The second loss sub-function is determined based on the calculation results of the regularization term of each of the multiple sample video frames.

[0021] Optionally, the sample character perspective video includes a plurality of character perspective video frames, and the expected character perspective video includes reference video frames corresponding to the plurality of character perspective video frames one by one; The determining of a first loss sub-function based on the sample character perspective video and the expected character perspective video includes: Determine a first pixel value of a pixel point of each of the character perspective video frames, and determine a second pixel value of a pixel point of each of the reference video frames; For each of the character perspective video frames, determining a pixel difference value based on the first pixel value and a second pixel value of the reference video frame corresponding to the character perspective video frame; The first loss sub-function is determined based on the pixel difference values ​​of each of the multiple character perspective video frames.

[0022] Optionally, the target model is obtained by adjusting the first model through the following steps: A reference video frame acquisition unit, used to acquire a plurality of reference video frames of a reference video; A convolution unit, configured to perform three-dimensional convolution processing on the multiple reference video frames to obtain predicted means and predicted standard deviations of the multiple reference video frames; A reparameterization unit, configured to reparameterize the plurality of reference video frames respectively based on the predicted mean and the predicted standard deviation to obtain video frame hidden state features of the plurality of reference video frames; A feature decoding unit, configured to decode the video frame hidden state features of each of the plurality of reference video frames to obtain a predicted video frame corresponding to each of the plurality of reference video frames; An adjustment unit is used to adjust the parameters of the first model based on the predicted video frame, the reference video frame, the predicted mean and the predicted standard deviation to obtain the target model.

[0023] Optionally, the adjustment unit is used to: Performing mean square error calculation based on the multiple reference video frames and the predicted video frames corresponding to the multiple reference video frames to obtain a video frame error function; Performing logarithmic calculation based on the predicted mean and the predicted standard deviation to obtain a log-likelihood function; Based on the video frame error function and the log-likelihood function, parameters of the first model are adjusted to obtain the target model.

[0024] Optionally, the video script description generation model is trained in the following manner: A reference video acquisition unit, configured to acquire a plurality of reference video frames arranged in time sequence of a reference video, and reference script description information corresponding to each of the plurality of reference video frames; a feature extraction unit, configured to extract reference video frame features of each of the plurality of reference video frames; A script generation unit, configured to generate a script by using a preset second model based on the reference video frame features of each of the multiple reference video frames and the reference script description information, to obtain predicted script information of each of the multiple reference video frames; The second training unit is used to train the second model based on the reference script description information and the predicted script information of each of the multiple reference video frames to obtain the video script description generation model.

[0025] Optionally, the script generation unit is used to: For each of the reference video frames, embedding the reference script description information to obtain a reference script description embedding feature; Based on the reference script description embedding feature, performing self-attention calculation through the second model to obtain a first attention calculation result; Based on the first attention calculation result and the reference video frame feature, performing cross attention calculation through the second model to obtain a second attention calculation result; Based on the second attention calculation result and the preset vocabulary in the second model, a script description is generated to obtain the predicted script information of the reference video frame.

[0026] Optionally, the second training unit is used for: For each of the reference video frames, based on the reference video frame script description information and the predicted script description information, determine a third loss sub-function; Determine a second loss function based on the plurality of third loss sub-functions; Based on the second loss function, the second model is trained to obtain the video script description generation model.

[0027] Optionally, the predicted script description information includes a first probability and a first sequence number of each of the plurality of predicted words, and the reference video frame script description information includes a second sequence number of each of the plurality of reference words; The determining, for each of the reference video frames, a third loss sub-function based on the reference video frame script description information and the predicted script description information comprises: For a single predicted word, if the predicted word is consistent with the reference word whose second serial number is the same as the first serial number of the single predicted word, determining the predicted word as a target word; The third loss sub-function is determined based on the negative logarithm of the first probability of the target word.

[0028] According to one aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method for generating a character perspective video when executing the computer program.

[0029] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a character perspective video as described above is implemented.

[0030] According to one aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program. The computer program is read and executed by a processor of an electronic device, so that the electronic device executes the character perspective video generation method as described above.

[0031] In the disclosed embodiment, source script information with a target role is extracted from a source video. The source script information is described from the perspective of a third party other than any role in the source script information. The source background description information reflects the background in which the source script exists, and the psychological activities and actions generated by any role are closely related to the background. With the source script information and the source background description information, the source script can be adjusted using the role script conversion model, and the source script information from the third party's perspective can be converted into the role script information from the perspective of the target role using the role script conversion model. The role script information reflects the adjustment of the psychological activities, actions generated, and pictures seen by the role under the source background description information. Then, based on the role script information, the picture information corresponding to the target role, and the source background description information, diffusion guidance information is generated, and based on the diffusion guidance information, a role perspective video from the perspective of the target role is generated. The picture information corresponding to the target role concretizes the image of the role in the role perspective video, and the source background description information determines the background and color tone of the picture in the role perspective video. Therefore, the specific video is generated by integrating the role script information, the role picture information and the source background description information, taking into account the influence of the role's appearance and the background of the picture on the specific content in the video, and improving the accuracy of the generated role perspective video, thereby improving the information transmission efficiency of the role perspective video. The disclosed embodiment can obtain the role perspective video of the target role perspective from the source video of the third party perspective with high accuracy.

[0032] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0034] Figure 1 is a system architecture diagram of a system to which the method for generating a character perspective video according to an embodiment of the present disclosure is applied; Figure 2A-2E A schematic diagram showing the application of the method for generating a character perspective video according to an embodiment of the present disclosure in a video production scenario; Figure 3A-3B A schematic diagram showing the application of the method for generating a character perspective video in a video playback scenario according to an embodiment of the present disclosure; Figure 4 is a flowchart of a method for generating a character perspective video according to an embodiment of the present disclosure; Figure 5 is a flowchart of extracting source script information according to an embodiment of the present disclosure; Figure 6 is a flow chart of generating video frame script description information according to an embodiment of the present disclosure; Figure 7 is a flow chart of generating target video frame features of a source video frame according to an embodiment of the present disclosure; Figure 8 is a flow chart of generating source script information according to video frame script description information according to an embodiment of the present disclosure; Fig. 9 It is a schematic diagram of the overall implementation process of extracting source script information according to an embodiment of the present disclosure; Fig.10 It is a schematic diagram of the implementation process of generating role script information according to an embodiment of the present disclosure; Fig.11 is a flow chart of generating role script information according to an embodiment of the present disclosure; Fig.12 is a flow chart of generating character script line information and character script scene information according to an embodiment of the present disclosure; Fig.13 It is a schematic diagram of the implementation process of generating character script line information and character script scene information according to an embodiment of the present disclosure; Fig.14 is a flow chart of generating diffusion guidance information according to an embodiment of the present disclosure; Fig.15 is a schematic diagram of an implementation process of generating a character perspective video using a diffusion model according to an embodiment of the present disclosure; Fig.16 is a schematic diagram of the overall implementation process of generating a character perspective video according to an embodiment of the present disclosure; Fig.17 is a flow chart of training a diffusion model according to one embodiment of the present disclosure; Fig.18 is a flow chart for generating a sample character perspective video according to one embodiment of the present disclosure; Fig.19 is a flow chart for generating sample video frame denoising features according to one embodiment of the present disclosure; Fig. 20 is a schematic diagram of an implementation process of generating sample video frame denoising features according to an embodiment of the present disclosure; Fig.21 is a flow chart of generating sample video frame denoising features according to another embodiment of the present disclosure; Fig. 22 is a flowchart of training a target model based on a first loss function according to an embodiment of the present disclosure; Fig.23 is a flow chart of adjusting parameters of a first model according to an embodiment of the present disclosure; Figure 24A-24C is a schematic diagram of the implementation process of the training diffusion model of an embodiment of the present disclosure; Fig.25 is a flow chart of a training video script description generation model according to one embodiment of the present disclosure; Fig.26 is a flowchart of generating predicted script information of a reference video frame according to an embodiment of the present disclosure; Fig. 27 is a module diagram of a device for generating a character perspective video according to an embodiment of the present disclosure; Fig.28 is a terminal structure diagram of a method for generating a character perspective video according to an embodiment of the present disclosure; Fig.29 It is a server structure diagram of a method for generating a character perspective video according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0036] The following describes the system architecture and scenarios for application of the embodiments of the present disclosure.

[0037] Figure 1 1 is a system architecture diagram used by the method for generating a character perspective video according to an embodiment of the present disclosure, and includes an object terminal 140, the Internet 130, a gateway 120, a video processing server 110, and a database 150, etc.

[0038] The object terminal 140 includes various forms such as desktop computers, laptop computers, PDAs (personal digital assistants), tablet computers, mobile phones, vehicle-mounted terminals, home theater terminals, smart TVs, and dedicated terminals. In addition, it can be a single device or a collection of multiple devices. The object terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data. The object terminal 140 includes a multimedia platform, which is used to play the film and television works selected by the object. The multimedia platform is also used to receive the film and television works selected by the object, the video clips of the film and television works, and the background description information, and submit the video clips and background description information of the film and television works to the video processing server, so that the video processing server generates a character perspective video with a certain character in the film and television works as the main perspective.

[0039] The video processing server 110 refers to a computer system that can provide certain services to the object terminal 140. Compared with the ordinary object terminal 140, the video processing server 110 has higher requirements in terms of stability, security, performance, etc. The video processing server 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), or a cloud server. The video processing server 110 includes various types of services, wherein the implementation of each service of the video processing server 110 is often associated with some intermediate databases or storage media. The video processing server 110 is used to generate a video script corresponding to the video clip according to the video clip of the film and television work selected by the object, and generate a supplementary script for a certain role using the video script and background description, and generate a role perspective video with the perspective of the role as the main perspective according to the background description submitted by the object, the supplementary script, and the role image of the role extracted from the database 150. The database 150 is used to store various images such as scene images of film and television works, character images of various characters, etc., wherein the database 150 can be set up separately or integrated in the video processing server 110 or other electronic devices.

[0040] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the object terminal 140 to the video processing server 110 is sent to the corresponding server through the gateway 120. The message sent by the video processing server 110 to the object terminal 140 is also sent to the corresponding object terminal 140 through the gateway 120.

[0041] The embodiments of the present disclosure can be applied in various scenarios, such as Figure 2A-2E The video production scene shown, Figure 3A-3B The video playback scene shown, etc.

[0042] 1. Video production scene.

[0043] like Figure 2AAs shown, when the subject wants to generate a role perspective video for a certain role in a certain film or TV work, the subject will log in to the multimedia platform on the subject terminal and enter the video production process. At this time, a prompt field "Please provide the following information" will be displayed on the page, and an editing area for entering the name of the film or TV series, an editing area for selecting a film or TV series clip, an editing area for entering a brief description of the film or TV series, an editing area for selecting the role corresponding to the target perspective, and an editing area for selecting the video storage location will be provided. Based on this, the object enters "Xxxx1" in the editing area for entering the name of the film and television drama; the object selects "D:\Film and Television Drama Xxxx1\Clip 1" in the editing area for selecting the film and television drama clip; the object enters "In the xxx era, character Axxx, character Bxx, xxxx" in the editing area for entering a brief description of the film and television drama, selects "Role B" in the editing area for selecting the character corresponding to the target perspective, selects "D:\Film and Television Drama Xxxx1\Character Video" in the editing area for selecting the video storage location, and clicks the "OK" button to determine the generation of a character perspective video with character B as the main perspective for clip 1 of the film and television drama Xxxx1, and store the generated character perspective video under the path "D:\Film and Television Drama Xxxx1\Character Video".

[0044] like Figure 2B As shown, after the subject clicks the "OK" button, a prompt window will be displayed on the page, wherein the prompt window has a prompt field "The character image of character B is being extracted from the database, please wait patiently..." to indicate to the subject the extraction process of the character image for character B before the video is generated.

[0045] like Figure 2C As shown, after the character images are extracted, a prompt window will be displayed on the page, wherein the prompt window has a prompt field "10 character images of character B have been extracted, and the supplementary character script of character B is being generated, please wait patiently..." to indicate the extraction results of the character images to the subject.

[0046] like Figure 2D As shown, it is a schematic diagram of the multimedia platform generating a supplementary role script for character B in video generation. In the page, a prompt field is displayed, "The supplementary role script corresponding to character B in clip 1 has been generated, see the attached figure for details. In addition, the generated supplementary role script has been saved to the path: M:\Film and Television Scripts\Supplementary Role Scripts.", and the supplementary role script is displayed as "(It is late at night, and character B is the only one in the company's conference room. He sits at the table with project resources spread out in front of him, looking tired and solemn.); Character B: (talking to himself) What went wrong? Why is this happening? (He flips through the information, trying to find the source of the error)", to indicate to the subject the generated supplementary role script for character B.

[0047] like Figure 2E As shown, after the supplementary role script is generated, a prompt window will be displayed on the page, wherein the prompt window has a prompt field "Generating the role perspective video of role B, please wait patiently..." to indicate the generation process of the role perspective video to the subject.

[0048] (2) Video playback scenarios.

[0049] like Figure 3A As shown, the subject is using the multimedia application on the subject's terminal to play a video clip of a film and television work, which is Episode 01 of the film and television work named "xxx Season 2". The total length of the video clip is 47:21, and at this time, the video clip is currently played to the 37th second. The video frame at the 37th second is a video screen displayed from a third-party perspective, with characters 1, 2, and 3 located on the left side of the video screen, characters 4, 5, and 6 located on the right side of the video screen, and character 7 located directly in front of the video screen. Based on this, when the subject wants to switch the video screen from the third-party perspective to the perspective of character 1, the subject will click the "Multi-Perspective" control below the video screen, and select "Role 1 Perspective" from the multiple character perspectives displayed by triggering the "Multi-Perspective" control to switch the video screen from the third-party perspective to the perspective of character 1.

[0050] like Figure 3B As shown, when the subject selects "Perspective of Character 1", the video screen will switch from the third-party perspective to the perspective of Character 1. At this time, the video screen is that Character 7 is displayed on the left side of the video screen, and Characters 4, 5 and 6 are displayed in front of the video screen.

[0051] It should be noted that the character perspective video generation method of the embodiment of the present disclosure can be applied not only to the above-mentioned video production scenarios and video playback scenarios, but also to various application scenarios such as video editing scenarios in self-media and personalized video production scenarios in social media.

[0052] The following is a general description of the embodiments of the present disclosure.

[0053] According to an embodiment of the present disclosure, a method for generating a character perspective video is provided.

[0054] The character perspective video generation method is generally used in business scenarios where auxiliary videos viewed from different character perspectives need to be generated for film and television works, such as Figure 2A-2E The video production scene shown, Figure 3A-3BThe video playback scene shown, etc. The disclosed embodiment provides a solution for obtaining a character perspective video from the perspective of a target character from a source video from a third-party perspective by integrating character script information, character picture information, and source background description information, which can improve the accuracy of generating auxiliary videos from different character perspectives for film and television works.

[0055] like Figure 4 As shown, the character perspective video generation method according to an embodiment of the present disclosure can be executed by an electronic device, which can be Figure 1 The video processing server or object terminal shown. A method for generating a role perspective video according to an embodiment of the present disclosure may include: Step 410: Obtain a source video having a target character and source background description information of the source video; Step 420: extracting source script information from the source video; Step 430: Obtaining the introductory text template information corresponding to the role script conversion model; Step 440: Generate role script information from the perspective of the target role using a role script conversion model based on the source script information, the source background description information and the guide template information; Step 450: Generate diffusion guidance information based on the role script information, the image information corresponding to the target role, and the source background description information, and generate a role perspective video from the perspective of the target role based on the diffusion guidance information.

[0056] Steps 410 - 450 are described in detail below.

[0057] In step 410, a source video having a target character and source background description information of the source video are obtained.

[0058] The target character refers to a character in the source video, where the target character can be a person or an animal in the source video.

[0059] The source video refers to the original video data of the film or television work for which the character's perspective video needs to be generated.

[0060] The source background description information is used to briefly introduce the background of the entire story of the source video in text form. The source background description information is generally a simplified version of the entire video script of the source video, often only including key plots, historical background, location, cultural environment or summary information.

[0061] When this embodiment is implemented, since the source video is often determined and published by the producer of the source video, based on this, with authorization and permission, the source video can be downloaded from official channels or public platforms to obtain the source video with the target character.

[0062] Similarly, when producing and publishing the source video, in order to let the public know the general outline of the source video, the producer of the source video often determines the source background description information of the source video and publishes the source background description information of the source video. Based on this, with authorization, the source background description information of the source video can be obtained from official channels or credible public platforms.

[0063] In step 420, source script information is extracted from the source video.

[0064] The source script information refers to the script content that the source video relies on when shooting. The source script information often includes scene descriptions, lines and actions of each character, and so on.

[0065] To save space, the specific process of extracting source script information from the source video in the embodiment of the present disclosure will be described in detail below and will not be repeated here.

[0066] In step 430, the introductory language template information corresponding to the role-playing script conversion model is obtained.

[0067] The role-script conversion model refers to a large language model that can automatically generate a role script for a target character based on information such as video clips, background descriptions, and preset prompts of film and television works.

[0068] The role-playing script conversion model in the disclosed embodiment may be composed of one language model or a collection of multiple language models, without limitation.

[0069] The introductory language template information is used to guide the role script conversion model to generate the framework of the role script information of the target role. The introductory language template information often includes prompt words related to the role name, emotional state, background information, scene setting, and plot development.

[0070] When this embodiment is implemented, due to different task requirements, the prompt information required by the role script conversion model to generate a specific text is different. In order to improve the efficiency of task processing, corresponding introductory language templates are often pre-set for various tasks, and multiple introductory language templates are stored in a template library. Based on this, with authorization, according to the task requirements of generating the role script information of the target role, the introductory language template information corresponding to the role script conversion model can be directly called from the template library.

[0071] In step 440, based on the source script information, the source background description information and the guide template information, the role script conversion model is used to generate the role script information from the perspective of the target character.

[0072] The role script information is used to indicate the script content generated to supplement the video content of the source video, with the target character's perspective as the main perspective. The role script information includes but is not limited to some possible plot developments, interactions between characters, etc. that are not in the source video.

[0073] To save space, the specific process of generating the role script information from the perspective of the target role based on the source script information, source background description information and guide template information in the embodiment of the present disclosure will be described in detail below and will not be repeated here.

[0074] In step 450, diffusion guidance information is generated based on the role script information, the picture information corresponding to the target role, and the source background description information, and a role perspective video from the perspective of the target role is generated based on the diffusion guidance information.

[0075] Among them, the step of generating a character perspective video from the perspective of the target character based on the diffusion guidance information may include but is not limited to: guiding the diffusion model based on the diffusion guidance information to generate a character perspective video from the perspective of the target character.

[0076] The diffusion guidance information is used to guide the video generation process of the diffusion model. The diffusion guidance information is used as a text condition constraint to assist the diffusion model in performing video frame denoising during video generation, so that the video frame effect meets the actual requirements.

[0077] The diffusion model (SD) of the embodiment of the present disclosure is a stable diffusion model from text to video. The diffusion model of the embodiment of the present disclosure often takes the background description of the film and television work, the text description of the character image, and the supplementary generated character script information as input, and outputs a video with the target character's perspective as the main perspective. The diffusion model can achieve the goal of obtaining the target character's perspective video from the source video of the third party's perspective.

[0078] A character perspective video refers to a video that uses the target character's perspective as the main perspective. The plot described in the character perspective video is similar to the plot described in the source video. The difference is that the character perspective video is presented from the perspective of the target character, while the source video is presented from a third-party perspective.

[0079] To save space, the specific process of generating diffusion guidance information based on role script information, image information corresponding to the target role, and source background description information, as well as the specific process of generating a role perspective video based on the diffusion guidance information to guide the diffusion model will be described in detail below and will not be elaborated here.

[0080] Through the above steps 410-450, the embodiment of the present disclosure extracts source script information with a target role from a source video. The source script information is described from the perspective of a third party other than any character in the source script information. The source background description information reflects the background in which the source script exists, and the psychological activities and actions generated by any character are closely related to the background. With the source script information and the source background description information, the source script can be adjusted using the role script conversion model, and the source script information from the perspective of the third party can be converted into the role script information from the perspective of the target character using the role script conversion model. The role script information reflects the adjustment of the psychological activities, actions generated, and pictures seen by the character under the source background description information. Then, based on the role script information, the picture information corresponding to the target character, and the source background description information, diffusion guidance information is generated, and based on the diffusion guidance information, a role perspective video from the perspective of the target character is generated. The picture information corresponding to the target character makes the image of the character in the role perspective video concrete, and the source background description information determines the background and color tone of the picture in the role perspective video. Therefore, the specific video is generated by integrating the role script information, role picture information and source background description information using the diffusion model, taking into account the influence of the role's appearance and the background of the picture on the specific content in the video, improving the accuracy of the generated role perspective video, thereby improving the information transmission efficiency of the role perspective video. The disclosed embodiment can obtain the role perspective video of the target role perspective from the source video of the third party perspective with high accuracy.

[0081] In addition, according to the embodiments of the present disclosure, the subject can understand the story of the film or television work to which the source video and the character perspective video belong from the perspective of different characters that the subject prefers, which can deepen the subject's understanding and involvement in the plot and increase the subject's immersion in watching these narrative film or television works, thereby increasing the user stickiness of the film or television works.

[0082] The above is a general description of steps 410 to 450. Since steps 410 and 430 have been described in detail in the above general description, the specific implementation of steps 420, 440 and 450 will be described in detail below.

[0083] Step 420 is described in detail below.

[0084] In step 420, source script information is extracted from the source video.

[0085] Please refer to Figure 5 In one embodiment, step 420 specifically includes but is not limited to the following steps 510-530: Step 510: segment the source video to obtain a plurality of source video frames arranged in time sequence; Step 520: Input the plurality of source video frames into a pre-trained video script description generation model for description generation, and obtain video frame script description information of the plurality of source video frames; Step 530: Integrate the video frame script description information of each of the multiple source video frames to obtain source script information.

[0086] Steps 510 - 530 are described in detail below.

[0087] In step 510, the source video frame refers to a video segment in the source video.

[0088] In the specific implementation of this embodiment, first, the video frame length of the source video frame to be segmented is set. Then, according to the set video frame length, a command line tool is used to segment the source video into multiple source video frames arranged in time sequence, and the multiple source video frames are respectively saved in image form, wherein the command line tool of the embodiment of the present disclosure can be FFmpeg.

[0089] In step 520, the video script description generation model refers to a neural network model that can generate a video script corresponding to the video of the film and television work. The video script description generation model of the embodiment of the present disclosure often takes the source video of the film and television work as input and outputs the script content corresponding to the source video of the film and television work.

[0090] Furthermore, the video script description generation model of the embodiment of the present disclosure may be a deep learning model (Vision Transformer, ViT) based on a transformer architecture.

[0091] The video frame script description information is used to indicate the script content corresponding to a single source video frame.

[0092] To save space, the specific process of inputting multiple source video frames into a pre-trained video script description generation model for description generation in the embodiment of the present disclosure will be described in detail below and will not be repeated here.

[0093] In step 530, in terms of time sequence, the video frame script description information of each of the multiple source video frames is spliced ​​in sequence to achieve integration of the multiple video frame script description information to obtain the source script information.

[0094] The benefit of this embodiment is that when generating source script information, the source video is first divided into multiple source video frames, and the script is generated with the source video frames as the basic unit, which can achieve fine-grained processing of the source video and make the generated source script information more accurate. Furthermore, when specifically generating the video frame script description information corresponding to each source video frame, it is achieved by relying on a pre-trained video script description generation model, which can improve the accuracy and efficiency of generating video frame script description information to a certain extent, and thus improve the degree of automation of character video generation.

[0095] In the disclosed embodiment, the video script description generation model includes a feature extraction network and a script description generation network. The feature extraction network is used to extract video feature information of each source video frame of the source video. The script description generation network is used to generate script description content corresponding to the source video frame according to the video feature information of each source video frame.

[0096] Please refer to Figure 6 In one embodiment, step 520 specifically includes but is not limited to the following steps 610-620: Step 610: For each source video frame, extract features of the source video frame based on a feature extraction network to obtain target video frame features of the source video frame; Step 620: Based on the target video frame features, a script description generation network is used to generate a script to obtain video frame script description information of the source video frame.

[0097] Steps 610-620 are described in detail below.

[0098] In step 610, the target video frame feature is used to indicate video feature information contained in a single source video frame.

[0099] When this embodiment is implemented, the feature extraction network may be a convolutional neural network structure. Specifically, for each source video frame, the source video frame is input into the feature extraction network, and multiple convolutional layers of the feature extraction network sequentially extract features from the source video frame, each convolutional layer extracts video feature information of the source video frame at different levels, and the output result of the last convolutional layer is used as the target video frame feature of the source video frame.

[0100] It should be noted that when multiple convolutional layers of the feature extraction network extract features from the source video frame in sequence, the input of the first convolutional layer is the source video frame, and the output is the video frame features corresponding to the source video frame; the input of the second convolutional layer is the video frame features output by the first convolutional layer, and the output is the video frame features, and so on. The video frame features output by the second to last convolutional layer are used as the input of the last convolutional layer, and the last convolutional layer outputs the target video frame features.

[0101] In step 620, the script description generation network can be a Transformer-based decoder structure, wherein the script description generation network is composed of an attention layer and a feed-forward layer.

[0102] In the specific implementation of this embodiment, for each source video frame, the target video frame features are input into the script description generation network. The attention layer of the script description generation network first performs self-attention calculation on the target video frame features to obtain the video frame attention features, and then the feedforward layer performs feedforward processing on the video frame attention features to achieve nonlinear transformation of the video frame attention features to obtain the transformed video frame features. Finally, the preset function of the script description generation network calculates the probability of each feature element in the transformed video frame feature pointing to each reference word in the preset vocabulary, and uses the reference word with the highest probability as the predicted word corresponding to the feature element, and splices the predicted words corresponding to each feature element together, thereby obtaining the video frame script description information of the source video frame. Among them, the preset function is generally an activation function such as a softmax function that can realize feature classification. The prediction vocabulary is a list consisting of multiple reference words that is pre-set according to business needs.

[0103] Among them, when using the attention layer to perform self-attention calculation on the target video frame features, the target video frame features are first linearly projected, and the query channel of the attention layer is used to project the video frame features into query features, the key channel of the attention layer is used to project the video frame features into key features, and the value channel of the attention layer is used to project the video frame features into value features. Then, the query feature and the transposed result of the key feature are inner-product processed to obtain the inner-product result, and the inner-product result is divided by the square root of the feature dimension of the key feature to obtain the attention score. Furthermore, the softmax function is used to convert the attention score into an attention weight, and the attention weight and the value feature are multiplied to obtain the video frame attention feature.

[0104] The benefit of this embodiment is that the script is generated by using the video script description generation model, and the automatic feature extraction and script description generation can significantly improve the generation efficiency of video frame script description information, and reduce the time cost of manual editing and creation. At the same time, the target video frame features of each source video frame are extracted by the feature extraction network, which can capture the spatial features (such as edges, textures, colors, etc.) in the source video frame, improve the consistency of the feature description of each source video frame in quality, and reduce the errors of human operation; at the same time, the attention operation and feedforward operation are performed on the extracted target video frame features, which can capture the dependency relationship between the various feature elements in the source video frame for analysis, and at the same time, the prediction words corresponding to each feature element are generated in combination with the probability calculation method, which can improve the accuracy of the generated video frame script description information.

[0105] Since the target video frame features in the above embodiment are the output of the last convolutional layer of the feature extraction network, this will make the feature information of the target video frame features not comprehensive enough, which will affect the accuracy of the generated video frame script description information. Based on this, the embodiment of the present disclosure provides a solution for extracting features from source video frames based on feature fusion, which can improve the feature comprehensiveness and feature richness of the target video frame features.

[0106] Please refer to Figure 7 In one embodiment, step 610 specifically includes but is not limited to the following steps 710-740: Step 710: for each source video frame, perform multi-scale downsampling on the source video frame based on a feature extraction network to obtain a first downsampling feature, a second downsampling feature, and a third downsampling feature; Step 720: Perform dimensionality reduction processing on the first down-sampled feature, the second down-sampled feature, and the third down-sampled feature, respectively, to obtain a first dimensionality reduction feature corresponding to the first down-sampled feature, a second dimensionality reduction feature corresponding to the second down-sampled feature, and a third dimensionality reduction feature corresponding to the third down-sampled feature; Step 730: performing splicing processing on the first dimensionality reduction feature, the second dimensionality reduction feature, and the third dimensionality reduction feature to obtain an initial video frame feature of the source video frame; Step 740: perform splicing processing on the initial video frame features of the source video frame and the initial video frame features of the previous source video frame of the source video frame to obtain the target video frame features of the source video frame.

[0107] Steps 710-740 are described in detail below.

[0108] In step 710, the first down-sampling feature, the second down-sampling feature and the third down-sampling feature respectively indicate video feature information of the source video frame at different scales, wherein there are certain differences in feature information contained in the first down-sampling feature, the second down-sampling feature and the third down-sampling feature.

[0109] Among them, the feature dimensions of the first down-sampling feature, the second down-sampling feature and the third down-sampling feature increase successively.

[0110] In the specific implementation of this embodiment, for each source video frame, multiple convolutional layers based on the feature extraction network sequentially perform convolution processing on the source video frame features, and each convolutional layer takes the output features of the previous convolutional layer as input and outputs a new video frame feature. Based on this, three convolutional layers can be selected from the multiple convolutional layers as target layers, and the output features of the three target layers are respectively used as the first down-sampling feature, the second down-sampling feature, and the third down-sampling feature.

[0111] For example, in the extraction network including six convolution layers, the feature size of the output feature of the first convolution layer is 1 / 2 of the video frame size of the source video frame; the feature size of the output feature of the second convolution layer is 1 / 4 of the video frame size of the source video frame; the feature size of the output feature of the third convolution layer is 1 / 8 of the video frame size of the source video frame; the feature size of the output feature of the fourth convolution layer is 1 / 16 of the video frame size of the source video frame; the feature size of the output feature of the fifth convolution layer is 1 / 32 of the video frame size of the source video frame; the feature size of the output feature of the sixth convolution layer is 1 / 64 of the video frame size of the source video frame. Based on this, the output feature of the third convolution layer is determined as the first down-sampling feature, the output feature of the fourth convolution layer is determined as the second down-sampling feature, and the output feature of the fifth convolution layer is determined as the third down-sampling feature.

[0112] It should be noted that the output features selected in the above examples are from three consecutive convolutional layers. In other specific examples, the extracted multiple down-sampled features may also come from multiple discontinuous convolutional layers.

[0113] In step 720, since the feature dimensions of the first down-sampled feature, the second down-sampled feature, and the third down-sampled feature are different. In order to fuse the feature information contained in the first down-sampled feature, the second down-sampled feature, and the third down-sampled feature, it is necessary to map the first down-sampled feature, the second down-sampled feature, and the third down-sampled feature to a preset one-dimensional vector space, respectively, and convert the first down-sampled feature, the second down-sampled feature, and the third down-sampled feature from a multi-dimensional array to a one-dimensional array, respectively, to obtain a first reduced dimensionality feature corresponding to the first down-sampled feature, a second reduced dimensionality feature corresponding to the second down-sampled feature, and a third reduced dimensionality feature corresponding to the third down-sampled feature. Among them, the first reduced dimensionality feature, the second reduced dimensionality feature, and the third reduced dimensionality feature are all one-dimensional feature vectors.

[0114] In step 730, the initial video frame features are used to indicate comprehensive feature information of the source video frame at multiple scales.

[0115] In the specific implementation of this embodiment, since the first dimensionality reduction feature, the second dimensionality reduction feature, and the third dimensionality reduction feature are features of the same feature dimension, based on this, the first dimensionality reduction feature, the second dimensionality reduction feature, and the third dimensionality reduction feature can be directly feature spliced ​​to obtain the initial video frame feature of the source video frame.

[0116] In step 740, the method of acquiring the initial video frame features of the previous source video frame is basically the same as steps 710-730, and will not be described in detail.

[0117] In the specific implementation of this embodiment, since the initial video frame feature of the source video frame and the initial video frame feature of the previous source video frame of the source video frame are features of the same feature dimension, based on this, the initial video frame feature of the source video frame and the initial video frame feature of the previous source video frame of the source video frame can be directly spliced ​​to obtain a spliced ​​feature with a longer feature length, and the obtained spliced ​​feature is used as the target video frame feature of the source video frame.

[0118] In other embodiments, vector addition may be directly performed on the initial video frame features of the source video frame and the initial video frame features of the previous source video frame to obtain the target video frame features of the source video frame.

[0119] It should be noted that if the current source video frame is the first source video frame of the source video, then when the initial video frame features of the source video frame and the initial video frame features of the previous source video frame of the source video frame are spliced ​​together, the initial video frame features of the previous source video frame of the source video frame can be set to zero, and the initial video frame features of the current source video frame can be directly used as the target video frame features of the source video frame.

[0120] The benefit of this embodiment is that, when extracting features from the source video frame, it takes into account the feature fusion of the output features of multiple convolutional layers of the feature extraction network, and at the same time, it takes into account that the feature dimensions of the output features of different convolutional layers are different and cannot be directly fused. The downsampled features of multiple scales are mapped to the same one-dimensional vector space, and the first dimension reduction feature, the second dimension reduction feature, and the third dimension reduction feature are spliced ​​into the initial video frame feature of the source video frame, which can improve the feature comprehensiveness and feature richness of the initial video frame feature. At the same time, since there is a dependency relationship between the source video frames in time sequence, the embodiment of the present disclosure also takes into account the splicing result of the initial video frame feature of the current source video frame and the initial video frame feature of the previous source video frame as the target video frame feature. When extracting, the feature dependency and feature correlation of the video frame features of each source video frame in time sequence are taken into account, which can better improve the comprehensiveness and accuracy of the feature information of the extracted target video frame feature.

[0121] Since the video script description generation model is often trained based on information such as videos and background descriptions corresponding to various categories of film and television works, the video script description generation model tends to meet the most basic requirements when generating video frame script description information, and often does not focus on the video frame script description needs in specific fields. However, video scripts for film and television works in different fields or categories often have different characteristics. Directly determining the source script information based on the video frame script description information generated by the video script description generation model will result in poor accuracy and coherence of the source script information. Based on this, a solution for polishing the video frame script description information is provided in the disclosed embodiments, which can improve the accuracy and coherence of the source script information.

[0122] Please refer to Figure 8 In one embodiment, step 530 specifically includes but is not limited to the following steps 810-820: Step 810, splicing the video frame script description information of each of the multiple source video frames to obtain preliminary script information; Step 820: polish the preliminary script information to obtain source script information.

[0123] Steps 810-820 are described in detail below.

[0124] In step 810, the preliminary script information is used to indicate that the overall script content corresponding to the source video is generated by the video script description generation model based on the video content of the source video.

[0125] When this embodiment is implemented, based on the time sequence of multiple source video frames, the script description information of multiple video frames is spliced ​​in sequence to form an overall description information, and the obtained overall description information is used as preliminary script information.

[0126] In step 820, in order to improve the polishing efficiency and polishing quality, when the preliminary script information is polished, the preliminary script information and the preset prompt words can be input into the preset large language model, and the large language model automatically polishes the preliminary script information. Among them, the preset prompt words can be "Please polish the following plot description to make it more coherent and fascinating: [plot description text]" and the like. Specifically, first, the preliminary script information is preprocessed, and the preprocessing steps include but are not limited to the following steps: removing redundant spaces in the preliminary script information, correcting punctuation marks, unifying the terms and styles in the entire preliminary script information, etc., to improve the consistency of the entire preliminary script information. Then, the preprocessed preliminary script information and the preset prompt words are input into the large language model together, and the large language model performs multiple iterations of polishing, and outputs a polished preliminary script information. Further, in order to make the script description conform to a specific style or requirement, the review end adds, deletes, and modifies the sentence structure and content details of the polished preliminary script information, so as to obtain the source script information with no logical errors in the plot description and coherent plot.

[0127] The advantage of this embodiment is that it takes into account the polishing of the video frame script description information corresponding to the source video frames, and also takes into account the overall coherence of the source script information. The video frame script description information corresponding to multiple source video frames is first spliced ​​into a whole to obtain preliminary script information, and then the preliminary script information is polished. Compared with polishing each video frame script description information separately, this allows the large language model to pay attention to the dependency and association of the video frame script description information during polishing, thereby improving the accuracy of the polishing process, and further making the generated source script information have better accuracy and coherence.

[0128] Combine the following Fig. 9 The specific process of extracting source script information from a source video in an embodiment of the present disclosure is described by way of example.

[0129] like Fig. 9As shown, for the source video frame in time window t, the multiple convolutional layers of the feature extraction network are first used to extract multi-scale features of the source video frame to obtain the first down-sampling feature, the second down-sampling feature and the third down-sampling feature; then, the first down-sampling feature, the second down-sampling feature and the third down-sampling feature are spliced ​​into the initial video frame feature, wherein the splicing process includes two steps of dimensionality reduction processing and splicing processing. Further, the initial video feature of the source video frame of time window t and the initial video frame feature of the source video frame of time window t-1 are spliced ​​to obtain the target video frame feature of the source video frame of time window t. Then, the target video frame feature is input into the script description generation network, and the target video frame feature is processed in turn through the attention layer and the feedforward layer to obtain the video frame script description information corresponding to the source video frame of time window t. Further, the video frame script description information of each of the multiple source video frames in the entire time interval (from 0 to t) is input into the large language model for polishing processing to obtain the role script information of the source video.

[0130] Step 440 is described in detail below.

[0131] In step 440, based on the source script information, the source background description information and the guide template information, the role script conversion model is used to generate the role script information from the perspective of the target character.

[0132] In the disclosed embodiment, the role-script conversion model may be a general large language model, wherein the role-script conversion model includes a first sub-model and a second sub-model.

[0133] The first sub-model is used to fill in the input guide template information according to the input source script information and source background description information to form a complete script prompt instance.

[0134] The second sub-model is used to automatically generate a complete role script based on the complete script prompt instance.

[0135] In one embodiment, step 440 specifically includes but is not limited to the following steps: Based on the guide template information, the source script information, and the source background description information, an instance is generated through the first sub-model to obtain a role script prompt instance; Based on the role script prompt instance, the script is generated through the second sub-model to obtain the role script information.

[0136] Among them, the role script prompt instance is used to indicate the overall outline content used to generate the role script information. The role script prompt instance is often composed of multiple parts, and the specific content of each part is extracted based on the source script information and the source background description information.

[0137] In the specific implementation of this embodiment, first, the introductory language template information, the source script information, and the source background description information are input into the role script conversion model, and the first sub-model first determines the main composition of the introductory language template information, divides the introductory language template information into multiple template modules, and determines the introductory language corresponding to each template module. Then, for each template module, according to the introductory language, the script elements corresponding to the introductory language are extracted from the source script information and the source background description information, and the extracted script elements are filled into the template module, thereby obtaining a role script prompt instance based on multiple template modules filled with script elements. Furthermore, the second sub-model automatically creates complete script elements such as role dialogue and plot development that meet the specific requirements of the role script prompt instance based on the role script prompt instance, and expands the role script prompt instance into a coherent and logical script text, thereby obtaining the role script information.

[0138] like Fig.10 As shown, it is a specific illustration of generating role script information based on the role script conversion model. Specifically, the role script conversion model is composed of 5 language models. Language model 1 can be a large language model ChatGPT developed by a certain subject, and language model 1 is mainly used to process the input source script information; language model 2 can be a language model Gemini developed by another subject, and language model 2 is mainly used to process the input source background description information; language model 3 can be a language model Claude, language model 4 can be a language model Coze, and language model 5 can be a language model Llama. Language models 3, 4 and 5 are mainly used to process the input guide template information. Therefore, after the guide template information, source script information, and source background description information are input into the role script conversion model, language model 3, language model 4 and language model 5 first generate guide words according to the guide template information, source script information, and source background description information. According to the guide words, language model 1 and language model 2 extract the script elements corresponding to the guide words from the source script information and the source background description information to generate the role script information.

[0139] The benefit of this embodiment is that when generating role script information, the first sub-model in the role script conversion model can automatically group the source script information and the source background description information according to the modules set in the guide template information and the guide words of each module, and fill the source script information and the source background description information into each module in the guide template information to form a script prompt instance corresponding to the source video. The script prompt instance is an overall outline for generating a script. Further, the second sub-model in the role script conversion model will automatically generate role script content with role dialogue, role psychological activity description, and scene description according to the role script prompt instance. This method generates role script information through two stages, allowing the role script prompt instance generated in the first stage to be modified and adjusted when generating role script information, thereby improving the flexibility and accuracy of role script information generation.

[0140] In a specific example, the preset introductory language template information can be represented as shown in Table 1. The role script prompt instance generated according to the introductory language template information, source script information and source background description information can be represented as shown in Table 2. The role script information generated according to the role script prompt instance can be represented as shown in Table 3.

[0141] Table 1-Introduction template information

[0142] Table 2 - Examples of role-playing prompts

[0143] Table 3-Role Script Information

[0144] Because different characters often have large differences in their lines and psychological activities in the same plot scene. If the role-script conversion model generates a role script without knowing the character personality, appearance and other image characteristics of the target character, it often results in the generated role script information not being consistent with the character characteristics of the target character. Based on this, in order to improve the accuracy of the generated role script information, the disclosed embodiment provides a solution for generating role script information based on character images, which can make the character lines and character actions in the role script information more in line with the character characteristics of the target character, thereby improving the accuracy of the generated role script information.

[0145] Please refer to Fig.11 In another embodiment, step 440 specifically includes but is not limited to the following steps 1110-1120: Step 1110, obtaining image information corresponding to the target character; Step 1120: Based on the source script information, the picture information, and the source background description information, the introductory language is generated using the introductory language template information, and the introductory language is input into the role script conversion model to obtain the role script information.

[0146] Steps 1110 - 1120 are described in detail below.

[0147] In step 1110, the image information is used to indicate the visual information of the target character in various scenes and various viewing angles, wherein the image information includes feature information such as makeup, facial expressions, and body movements of the target character in various scenes and various viewing angles.

[0148] In the specific implementation of this embodiment, since the clothing, hairstyle, expression and accessories of the target character that reflect the character characteristics are intuitively reflected in the character image of the target character, and the character image of the target character is often stored in a database, based on this, with authorization, the character images of the target character in different viewing angles and different scenes are extracted from the database, and the obtained multiple character images are integrated to obtain the picture information corresponding to the target character.

[0149] In step 1120, first, the source script information, the image information, and the source background description information are integrated to obtain integrated information. Then, the introductory phrase template information is used to generate a more specific introductory phrase in combination with the integrated information. Further, the generated introductory phrase is input into the role script conversion model, so that the role script conversion model generates role script information according to the introductory phrase and various knowledge learned by itself.

[0150] The advantage of this embodiment is that when generating role script information, the influence of the character image of the target character on the character lines and character actions of the target character is taken into consideration, and the picture information describing the character characteristics of the target character, the source script information, and the source background description information are input into the role script conversion model together, so that the generated guide words are more accurate. In this way, when generating a role script, the role script conversion model will customize the script according to the appearance characteristics of the target character, and based on the character's personality introduction, etc., use narration descriptions and other methods in the role script information to construct a more vivid and three-dimensional character image, so that the character lines and character actions in the generated role script information are more in line with the character characteristics of the target character, which can improve the accuracy of the generated role script information to a certain extent.

[0151] Since the role script information contains various types of script elements, such as role lines, scene descriptions, story backgrounds, role introductions, etc., all belong to the category of role script information. In order to generate various types of script elements in a targeted manner, the disclosed embodiment provides a solution for generating multiple types of script information in parallel, which can improve the generation accuracy of role script information.

[0152] In the disclosed embodiment, the source script information includes source line information and source scene information, the role script information includes role script line information and role script scene information, the introductory language template information includes first introductory language template information and second introductory language template information, and the introductory language includes first introductory language and second introductory language.

[0153] The source line information is used to indicate the dialogue lines and monologue lines of each character in the source video.

[0154] The source scene information is used to indicate the scene in which each plot in the source video occurs.

[0155] The character script line information is used to indicate the dialogue lines and monologue lines of each character in the character perspective video with the target character as the main perspective.

[0156] The character script scene information is used to indicate what the scene is like when each plot occurs in the character perspective video with the target character as the main perspective.

[0157] The first introductory language template information is used to indicate the specific structure and content of the introductory language for generating the character's lines.

[0158] The second guide phrase template information is used to indicate the specific structure and content of the guide phrase for generating the scene information.

[0159] The first guide is used to guide the role script conversion model to generate role dialogue information for a role perspective video with the target character as the main perspective.

[0160] The second guide is used to guide the role-script conversion model to generate scene information of a role-perspective video with the target character as the main perspective.

[0161] Please refer to Fig.12 In another embodiment, step 1120 specifically includes but is not limited to the following steps 1210-1220: Step 1210: Based on the source dialogue information, source scene information, picture information and source background description information, generate the first guide language using the first guide language template information, and input the first guide language into the role script conversion model to obtain the role script dialogue information; Step 1220: Based on the source scene information and the source background description information, generate the second guide words using the second guide word template information, and input the second guide words into the role-script conversion model to obtain the role-script scene information.

[0162] Steps 1210 - 1220 are described in detail below.

[0163] In step 1210, the first introductory language template information is used to generate introductory language according to the source line information, source scene information, picture information and source background description information, and information related to the character lines is extracted from the source line information, source scene information, picture information and source background description information and filled into each part of the first introductory language template to generate the first introductory language. Next, the first introductory language is input into the role script conversion model, so that the role script conversion model generates the role script line information according to the first introductory language using its own corpus basis.

[0164] The specific process of step 1220 is similar to the above step 1210. The difference is that the generation of the role script scene information does not depend on the image information and the source line information, but the generation of the role script line information needs to depend on the source scene information. To save space, it will not be repeated.

[0165] like Fig.13 As shown, when generating role script information, the generation of role script lines information and role script scene information is performed simultaneously. Specifically, the first branch is to generate guide words based on source line information, source scene information, picture information and source background description information, using the first guide language template information, extracting information related to the role lines from the source line information, source scene information, picture information and source background description information and filling it into each part of the first guide language template, generating the first guide language, and inputting the first guide language into the role script conversion model, so that the role script conversion model generates role script lines information based on the first guide language. The second branch is to generate guide words based on source scene information and source background description information, using the second guide language template information, extracting information related to the scene from the source scene information and source background description information and filling it into each part of the second guide language template, generating the second guide language, and inputting the second guide language into the role script conversion model, so that the role script conversion model generates role script scene information based on the second guide language.

[0166] The benefit of this embodiment is that, since the role lines and scene information in the role script information are two relatively important parts, for these two different types of script information, two different introductory language template information are considered to be set respectively: the first introductory language template information suitable for generating the role script lines, and the second introductory language template information suitable for generating the role script scene; further, based on the first introductory language template information and the second introductory language template information, the role script conversion model is used to generate the role script line information and the role script scene information in parallel. At the same time, when generating the role script line information and the role script scene information, it is considered that there will be differences in the description information associated with the lines and the scenes, and the source line information, the source scene information, the picture information and the source background description information are used to generate the lines, and only the source scene information and the source background description information are used to generate the description information, so that while the source line information and the picture information are used to optimize the generation of the role lines, the interference of the source line information and the picture information on the generation of the role scene information is eliminated. This method can improve the generation accuracy of the role script line information and the role script scene information in the role script information to a certain extent.

[0167] Step 450 is described in detail below.

[0168] In step 450, diffusion guidance information is generated based on the role script information, the picture information corresponding to the target role, and the source background description information, and a role perspective video from the perspective of the target role is generated based on the diffusion guidance information.

[0169] Please refer to Fig.14 In one embodiment, the diffusion guidance information is generated by: Step 1410: Encode the source background description information to obtain background description encoding features; Step 1420, encode the role script information to obtain the role script encoding feature; Step 1430: extract features from the image information to obtain target character image features; Step 1440: Based on the background description coding features, the role script coding features, and the target role image features, conditional information is generated through a preset conditional fusion network to obtain diffusion guidance information.

[0170] Steps 1410-1440 are described in detail below.

[0171] In step 1410, the background description encoding feature is used to characterize the plot background of the source video in the form of a vector feature.

[0172] When this embodiment is implemented, the source background description information can be encoded based on a preset text encoder to obtain the role script encoding features.

[0173] In the embodiment of the present disclosure, the text encoder includes a tokenizer, an embedding layer, and a text attention calculation module (text transformer).

[0174] Specifically, first, the source background description information is input into a word segmenter, and the source background description information is segmented based on spaces, punctuation marks or separators based on the word segmenter to obtain multiple words. Furthermore, the stop words in the multiple words are removed to obtain multiple description words, and the index corresponding to each description word is found in a preset index table, and the description word is represented by the index. Among them, the preset index table is used to indicate the correspondence between words and indexes. Then, the index of each description word is input into the embedding layer for word embedding processing to obtain the description word embedding features corresponding to each description word. Finally, the description word embedding features of each description word are input into the text attention calculation module for attention calculation, and the text attention calculation module outputs the role script encoding features corresponding to the source background description information.

[0175] In step 1420, the role script encoding features are used to represent the supplementary script content that the target role needs to perform in the form of vector features.

[0176] When this embodiment is implemented, the specific process of step 1420 is similar to the above step 1410. To save space, it will not be repeated.

[0177] In step 1430, the target character image features are used to represent the character characteristics of the target character in the form of vector features, wherein the character characteristics include but are not limited to the expression characteristics, makeup characteristics, and action characteristics of the target character.

[0178] In the specific implementation of this embodiment, the image information may be subjected to feature extraction based on a preset image encoder to obtain the target character image features. The preset image encoder may be an image encoder based on a transformer structure (Vision Transformer, ViT).

[0179] Specifically, when extracting features from image information based on an image encoder, the image information is first divided into multiple image blocks of the same area. Next, the image encoder is used to perform attention calculation and feedforward processing on each image block to obtain the image block coding features corresponding to each image block. Finally, multiple image block coding features are spliced ​​into the target character image features. Among them, the specific process of attention calculation and feedforward processing is similar to the self-attention calculation and feedforward processing of step 620 above. The difference is that the input of the feedforward processing of step 1430 is the splicing result of the attention calculation result and the image block coding feature; and the input of the feedforward processing of step 620 is the video frame attention feature (self-attention calculation result). To save space, it will not be repeated.

[0180] In step 1440, the conditional fusion network refers to a neural network structure used to integrate multiple conditional constraint vectors into a whole.

[0181] In the specific implementation of this embodiment, the feature dimensions of the background description coding features, the role script coding features, and the target character image features may be different. In order to achieve the integration of multiple conditional constraints, first, the background description coding features, the role script coding features, and the target character image features are input into the conditional fusion network, and the conditional fusion network first maps the background description coding features, the role script coding features, and the target character image features to the same feature vector space to obtain the background description coding features, the role script coding features, and the target character image features with the same feature dimensions. Then, the conditional fusion network will use feature splicing or feature addition to fuse the background description coding features, the role script coding features, and the target character image features with the same feature dimensions, so as to integrate multiple conditional constraint vectors into a whole, thereby obtaining diffusion guidance information. Among them, the diffusion guidance information is often represented in the form of a vector.

[0182] The advantage of this embodiment is that the three kinds of feature information, namely, the role script information, the image information corresponding to the target role, and the source background description information, are integrated into a whole and used together as conditional constraints for denoising, which can improve the comprehensiveness and content effectiveness of the diffusion guidance information. This method takes into account the influence of the role's appearance and the background of the picture on the specific content in the video, so that under the guidance of the conditional guidance information, the diffusion model can generate a more accurate role perspective video.

[0183] In the disclosed embodiment, the diffusion model includes an encoding network, a decoding network, a diffusion network and a denoising network. Among them, the encoding network is used to encode the random noise map into a coding feature corresponding to the random noise map. The diffusion network is used to implement a step-by-step noise processing of the coding feature until the input coding feature approaches pure noise. The denoising network is used to gradually denoise the latent space feature vector output by the diffusion network under given conditional constraints until a denoising result that meets the conditional constraints is obtained. The decoding network is used to convert the character video frame features in the denoising result from the latent vector space to the pixel space, thereby generating a character perspective video.

[0184] In one embodiment, the process of generating a character perspective video from the perspective of a target character based on the diffusion guidance information guiding the diffusion model includes but is not limited to the following steps: The random noise image is encoded based on the encoding network to obtain the encoding features of the target video frame; Based on the diffusion network, the target video frame encoding features are diffused to obtain the target latent space feature vector; Based on the diffusion guidance information, the target latent space feature vector is denoised through the denoising network to obtain the video frame denoising feature; The denoising features of the video frames are decoded based on a decoding network to obtain a plurality of character video frames, and a character perspective video from the perspective of the target character is generated based on the plurality of character video frames.

[0185] Specifically, the specific process of generating the character perspective video of the target character perspective based on the diffusion guidance information guiding the diffusion model in this embodiment is similar to steps 1810-1840 of the embodiment of the present disclosure. The specific process of steps 1810-1840 will be described in detail below. To save space, it will not be repeated here.

[0186] It should be noted that the random noise map can be generated by a library function of a random number generator (for example, numpy.random.normal() function) according to the mean value of 0 and the standard deviation of 1 that the Gaussian distribution follows, and a random number that follows the Gaussian distribution is generated. Further, the random number that follows the Gaussian distribution is converted into the form of a random noise map, and the random noise map required by the diffusion model in video generation is obtained.

[0187] like Fig.15 , which is an overall flow chart of video generation based on a diffusion model according to an embodiment of the present disclosure. Specifically, the role script information, the picture information corresponding to the target role and the source background description information are used as inputs of the diffusion model.

[0188] First, a random noise image is generated using a random number i.

[0189] Next, the random noise image is encoded using the encoding network to obtain the target video frame encoding feature Z, and the target video frame encoding feature Z is diffused through the diffusion network. The target video frame encoding feature Z is denoised T times using the diffusion network to obtain the target latent space feature vector Z T , and its specific process is similar to the following step 1810. Further, the source background description information, role script information and picture information are used as conditional constraints, converted into a form that meets the input requirements of the conditional fusion network τ, and the source background description information, role script information and picture information are input into the conditional fusion network, and the conditional fusion network outputs diffusion guidance information, and its specific process is similar to the above steps 1410-1440.

[0190] Next, in the denoising network, the target latent space feature vector Z T When denoising, firstly, based on the diffusion guidance information, the target latent space feature vector Z is denoised through the downsampling module of the denoising network. T Down-sampling is performed to obtain the target down-sampling result. Then, based on the diffusion guidance information, the target down-sampling result is up-sampled through the up-sampling module of the denoising network to obtain the denoising result Z after one denoising operation. T-1' . Further, the denoising network is used to denoise the result Z at the prediction time step T T-1' According to the above process, denoising is performed. After T-1 denoising times, the video frame denoising feature Z is obtained. ’ Finally, the video frame denoising feature Z is constructed based on the decoding network. ’ Feature decoding is performed to obtain multiple character video frames I, and a character perspective video of the target character perspective is generated based on the multiple character video frames I. The specific process is similar to the following steps 1820-1840. To save space, it will not be repeated.

[0191] The benefit of this embodiment is that the target video frame encoding features corresponding to the random noise image are first denoised multiple times using a diffusion network to obtain a target latent space feature vector. Furthermore, the target latent space feature vector is denoised multiple times using a denoising network to achieve feature prediction of the character perspective video to be generated; in this process, the predicted video frame denoising features are fine-tuned using diffusion guidance information to achieve feature information adjustment at the image angle and text angle, so that the character lines, character scenes and other information in the final predicted denoising results are more accurate. Finally, the video frame denoising features are decoded using a decoding network to obtain a character perspective video from the target character's perspective. This method can obtain a character perspective video from the target character's perspective from a source video from a third-party perspective with high accuracy.

[0192] Combine the following Fig.16 The overall process of generating a character perspective video from the perspective of a target character based on a source video is described by way of example.

[0193] like Fig.16 As shown, when generating a role perspective video of the target role perspective for a source video, firstly, multiple source video frames of the source video are respectively input into the video script description generation model, and the feature extraction network and the script description generation network of the video script description generation model are used to extract features and generate script descriptions for the source video frames in turn, so as to obtain video frame script description information of each of the multiple source video frames, and the multiple video frame script description information are integrated into the source script information corresponding to the source video. Further, the source script information, the source background description information and the guide template information are input into the role script conversion model composed of language model 1, language model 2, language model 3, language model 4 and language model 5, and the role script conversion model outputs the role script information of the target role. Finally, the role script information, the image information corresponding to the target role and the source background description information are input into the diffusion model as conditional constraints (diffusion guidance information), and the encoding network, diffusion network, denoising network and decoding network of the diffusion network are used to generate the video, and finally the role perspective video corresponding to the source video is obtained, and the role perspective video is a video with the perspective of the target role as the main perspective.

[0194] The following is a detailed description of the training process of the diffusion model according to the embodiment of the present disclosure.

[0195] Please refer to Fig.17 In one embodiment, the diffusion model is trained by the following steps: Step 1710: Obtain a sample video having a sample role, sample background description information of the sample video, and an expected role perspective video of the sample role; Step 1720: extracting sample script information from the sample video; Step 1730: Generate sample role script information from the perspective of a sample role using a role script conversion model based on the sample script information, the sample background description information, and the preset introductory language template information; Step 1740: Generate sample guidance information based on the sample role script information, the sample picture information corresponding to the sample role, and the sample background description information; Step 1750: Based on the sample guidance information, generate a video through the target model to obtain a sample character perspective video from the sample character perspective; Step 1760: Based on the sample character perspective video and the expected character perspective video, train the target model to obtain a diffusion model.

[0196] Steps 1710-1760 are described in detail below.

[0197] In step 1710, the sample character refers to a character in the sample video, wherein the target character can be a person or an animal in the source video.

[0198] Sample videos refer to video data corresponding to certain narrative film and television works used to train diffusion models.

[0199] The sample background description information is used to briefly introduce the background of the entire story of the sample video in text form.

[0200] The expected character perspective video refers to a video that is expected to be generated with the perspective of the sample character as the main perspective for the sample video.

[0201] When this embodiment is implemented, the specific process of the sample video with the sample role and the sample background description information of the sample video in step 1710 is similar to the above step 410. To save space, it will not be repeated.

[0202] Furthermore, the expected character perspective video may be specially shot by the producer of the sample video, or may be pre-generated based on video editing software, etc., without limitation.

[0203] In step 1720, the sample script information refers to the script content that the sample video relies on when shooting.

[0204] In the specific implementation of this embodiment, the specific process of step 1720 is similar to the specific process of steps 510-530 described above, and will not be described in detail to save space.

[0205] In step 1730, the sample character script information is used to indicate the script content generated to supplement the video content of the sample video, with the sample character's perspective as the main perspective.

[0206] Among them, the preset introductory template information and the role script conversion model are basically consistent with those in the above step 430.

[0207] In the specific implementation of this embodiment, the specific process of step 1730 is similar to the specific process of steps 1110-1120 or steps 1210-1220, which will not be described in detail to save space.

[0208] In step 1740, the sample guidance information is used as a text condition constraint to assist the target model in performing video frame denoising during video generation, so that the feature information of the sample character video generated by the target model is as close as possible to the feature information of the expected character perspective video.

[0209] In the specific implementation of this embodiment, the specific process of step 1740 is similar to the specific process of steps 1410-1440 described above, and will not be described in detail to save space.

[0210] In step 1750, the target model is an untrained neural network model, which takes the background description of the film and television work, the text description of the character image, and the supplementary generated character script information as input, and outputs a neural network model of a video with the perspective of the sample character as the main perspective. Among them, the model structure of the target model is consistent with the model structure of the diffusion model, but there are certain differences in the model parameters of the target model and the diffusion model.

[0211] The sample character perspective video refers to a video that uses the sample character's perspective as the main perspective. The plot described in the sample character perspective video is similar to the plot described in the sample video. The difference is that the sample character perspective video is presented from the perspective of the sample character, while the sample video is presented from the perspective of a third party.

[0212] To save space, the specific process of generating a video based on the sample guide information and the target model to obtain a sample character perspective video of the sample character perspective in the embodiment of the present disclosure will be described in detail below.

[0213] In step 1760, the training goal is to minimize the difference between the sample character's perspective video and the expected character's perspective video. Through iterative training, the model parameters of the target model are continuously adjusted in each iterative training round, and the above steps 1710-1750 are continuously repeated until the difference between the sample character's perspective video and the expected character's perspective video is maximized in a certain iterative training round. The model parameters of the target model in the iterative training round are determined as the final model parameters, and the target model with the final model parameters is determined as the trained diffusion model.

[0214] The benefit of this embodiment is that it utilizes the expected role perspective videos pre-set for the sample roles of the sample videos to compare and evaluate the sample role perspective videos generated by the target model based on the conditional constraints of the sample role script information, sample image information and sample background description information. It is able to improve the learning and mining of video content from the perspective of a specific role by the target model through iterative training, improve the accuracy of the role perspective videos generated by the trained diffusion model from the perspective of a specific role, meet the video playback needs of different groups of people, and thereby improve the display diversity of video information of various film and television works and the information transmission efficiency of role perspective videos.

[0215] In the embodiment of the present disclosure, the sample video includes a plurality of sample video frames, wherein the method of acquiring the plurality of sample video frames is substantially the same as that of the above step 510. To save space, it will not be described here in detail.

[0216] At the same time, the target model in the embodiment of the present disclosure includes a diffusion network, a denoising network, and a decoding network. The specific introduction of the diffusion network, the denoising network, and the decoding network in the target model is basically consistent with the specific structure and specific introduction of the diffusion model mentioned above. In order to save space, it will not be repeated here.

[0217] Please refer to Fig.17 In one embodiment, step 1750 specifically includes but is not limited to the following steps 1810-1840: Step 1810: for each sample video frame, diffusion processing is performed on the sample video frame encoding features of the sample video frame based on the diffusion network to obtain a sample latent space feature vector; Step 1820: Based on the sample guidance information, the sample latent space feature vector is denoised by a denoising network to obtain a sample video frame denoising feature; Step 1830: Decoding the denoising features of the sample video frames based on the decoding network to obtain predicted video frames of each sample video frame; Step 1840: Obtain a sample character perspective video based on the predicted video frames of each of the multiple sample video frames.

[0218] Steps 1810-1840 are described in detail below.

[0219] In step 1810 , the sample video frame encoding feature is used to indicate a feature representation of the sample video frame in a vector space (latent space).

[0220] The sample latent space feature vector is used to indicate the result of adding noise to the sample video frame encoding features at a fixed time step through the diffusion network.

[0221] In the specific implementation of this embodiment, first, with authorization, a preset image encoder is called. Then, the sample video frame is input into the image encoder, and the sample video frame is encoded based on the image encoder to achieve the mapping of the sample video frame from the pixel space to the vector space (latent space), thereby obtaining the encoding features of the sample video frame.

[0222] Furthermore, according to the preset time step, noise is added to the sample video frame coding features through the diffusion network one by one, so that the sample video frame coding features gradually lose their original features. When the number of noise additions reaches the preset time step, the sample video frame coding features will become a latent space representation without any features. At this time, the obtained latent space representation is determined as the sample latent space feature vector.

[0223] For example, the preset time step is set to T, where T is a positive integer, and the sample video frame coding features are denoised T times using the diffusion network, and the result after T times of denoising is used as the sample latent space feature vector.

[0224] In step 1820, the sample video frame denoising feature is used to indicate the video frame feature that satisfies the constraint requirements of the sample guidance information, which is generated by denoising the sample latent space feature vector at a fixed time step through the denoising network.

[0225] In the specific implementation of this embodiment, the sample latent space feature vector is denoised one by one through the denoising network according to the preset time step, and in each denoising, the sample guidance information is used as a conditional constraint so that the denoising of the sample latent space feature vector meets the requirements of the sample guidance information. Further, in the T denoising process, except that the first denoising operation uses the sample latent space feature vector as the denoising object, the subsequent T-1 denoising operations all use the denoising result of the previous denoising operation as the denoising object, and the denoising result of the last denoising operation is determined as the sample video frame denoising feature.

[0226] In step 1830, the predicted video frame refers to a single video frame with the sample character's perspective as the main perspective, which is constructed by the decoding network based on the denoising features of the sample video frame.

[0227] In the specific implementation of this embodiment, for each sample video frame, the denoising features of the sample video frame are input into the decoding network, and the denoising features of the sample video frame are mapped from the latent vector space back to the original pixel space through the decoding network to generate a noise-free predicted video frame that is consistent with the sample guidance information.

[0228] In step 1840, based on the temporal sequence of the multiple sample video frames, the predicted video frames are spliced ​​in sequence according to this sequence, so that the multiple predicted video frames are spliced ​​into a whole, thereby obtaining a sample character perspective video.

[0229] The benefit of this embodiment is that the sample video frame is mapped from the pixel space to the latent space, so that the sample video frame becomes a sample video frame coding feature that meets the input requirements of the target model, which can improve the usability of the sample video frame. Furthermore, the sample video frame coding feature is denoised multiple times using the diffusion network of the target model, so that the pure noise processing of the sample video frame coding feature is realized, and the effective video frame features in the sample video frame coding feature can be effectively stripped, so that the model can perform feature recovery according to the sample guidance information in the subsequent denoising process, and decode the recovered sample video frame denoising feature into a predicted video frame, and generate a sample character perspective video according to the predicted video frame. This method is conducive to improving the processing accuracy of the model for the character perspective video generation task.

[0230] In the embodiment of the present disclosure, the denoising network may be a U-type network structure (U-net network), and the denoising network includes an upsampling module and a downsampling module.

[0231] The upsampling module is used to downsample the sample latent space feature vector to obtain more low-dimensional features.

[0232] The down-sampling module is used to perform up-sampling processing on the output result of the up-sampling module, and restore the output result to the sample video frame denoising feature with the same feature dimension as the sample latent space feature vector.

[0233] Please refer to Fig.19 In one embodiment, step 1820 specifically includes but is not limited to the following steps 1910-1930: Step 1910: performing splicing processing on the sample guidance information and the sample latent space feature vector to obtain a sample denoising splicing feature; Step 1920: downsample the sample denoising splicing features based on the downsampling module to obtain video frame downsampled features; Step 1930: Upsample the downsampled features of the video frame based on the upsampling module to obtain denoising features of the sample video frame.

[0234] Steps 1910-1930 are described in detail below.

[0235] In step 1910, the sample denoising splicing feature is used to characterize the feature representation of fusing the sample guidance information into the sample latent space feature vector.

[0236] In the specific implementation of this embodiment, the sample guidance information and the sample latent space feature vector are spliced ​​to obtain a feature vector with a longer feature length, and the spliced ​​feature vector is determined as the sample denoising splicing feature.

[0237] In step 1920, the video frame down-sampling feature is used to indicate the video frame feature generated by denoising the sample latent space feature vector by down-sampling.

[0238] In the specific implementation of this embodiment, in order to improve the learning effect of the model on the script information, background information and character image information in the sample guidance information during the downsampling process, a self-attention mechanism can be introduced in the downsampling process, wherein the self-attention calculation method in the downsampling process is similar to the self-attention calculation method in the above step 620. To save space, it will not be repeated.

[0239] In another embodiment, step 1820 may also introduce a cross-attention mechanism in the downsampling process. Specifically, first, linearly project the sample latent space feature vector to obtain a query vector, and linearly project the sample guidance information to obtain a key vector and a value vector. Next, a cross-attention calculation is performed based on the query vector, the key vector, and the value vector to obtain the video frame downsampling features. This method enables the target model to guide the generation process of the sample character perspective video according to the sample guidance information.

[0240] In step 1930, the specific process of step 1920 is similar to the specific process of step 1910. The difference is that step 1910 is for downsampling of the sample latent space feature vector; while step 1920 is for upsampling of the sample downsampling result, and the specific processes of the two are symmetrical. In addition, the feature dimension of the sample video frame denoising feature obtained in step 1920 is the same as the feature dimension of the sample latent space feature vector input in step 1920.

[0241] like Fig. 20As shown, it is a specific schematic diagram of the denoising network performing a denoising operation according to the sample guidance information (when the predetermined time step is T=1). Specifically, the denoising network includes a downsampling module and an upsampling module, wherein the downsampling module includes an attention downsampling module 1 having an attention downsampling layer and a residual block structure, and an attention downsampling module 2 having an attention downsampling layer and a residual block structure. The upsampling module includes an attention downsampling module 1 having an attention upsampling layer and a residual block structure, and an attention upsampling module 2 having an attention upsampling layer and a residual block structure. First, the attention downsampling layer in the attention downsampling module 1 performs attention calculation on the sample latent space feature vector based on the sample guidance information to obtain a first attention calculation result, and the residual block structure in the attention downsampling module 1 performs residual processing on the first attention calculation result to obtain a first downsampling result. Further, the attention downsampling layer in the attention downsampling module 2 performs attention calculation on the first downsampling result based on the sample guidance information to obtain a second attention calculation result, and the residual block structure in the attention downsampling module 2 performs residual processing on the second attention calculation result to obtain the video frame downsampling feature. Then, the spatial transformation structure performs spatial transformation on the video frame downsampling feature to obtain the spatially transformed video frame downsampling feature. Further, the attention upsampling layer in the attention upsampling module 2 performs attention calculation on the spatially transformed video frame downsampling feature based on the sample guidance information to obtain a third attention calculation result, and the residual block structure in the attention downsampling module 2 performs residual processing on the third attention calculation result to obtain the first upsampling result. Finally, the attention upsampling layer in the attention upsampling module 1 performs attention calculation on the first upsampling result based on the sample guidance information to obtain a fourth attention calculation result, and the residual block structure in the attention upsampling module 2 performs residual processing on the fourth attention calculation result to obtain the sample denoising splicing feature.

[0242] It should be noted that the above denoising operation is shown with T=1 as an example. When the predetermined time step is greater than 1, the first denoising operation is exactly the same as the above process. From the second denoising operation, the denoising result of the previous denoising operation is used to replace the sample latent space feature vector in the first denoising operation, and the same process is performed. At this time, the sample denoising splicing feature refers to the final output of the Tth denoising operation (the last denoising operation).

[0243] The benefit of this embodiment is that the denoising process is divided into a downsampling stage and an upsampling stage, and the upsampling stage and the downsampling stage rely on the same sample guidance information, which can improve the consistency of feature fine-tuning in the downsampling stage and the upsampling stage. Specifically, in the downsampling stage, the downsampling module performs feature downsampling according to the sample guidance information, and in the upsampling stage, the upsampling module performs feature upsampling according to the sample guidance information. In this way, in both the downsampling stage and the upsampling stage, the three aspects of control information, namely, the role script information, the role picture information, and the source background description information, are combined for guidance and constraints, which can achieve accurate fine-tuning and flexible control of video frame denoising. This method enables the model to perform denoising processing under various conditions and constraints, improves the denoising ability of the model, and is conducive to the model to construct specific content in the video according to the appearance of the character and the background of the picture, thereby improving the accuracy of the model generating the role perspective video.

[0244] Since the denoising network uses the sample guidance information as a conditional constraint each time it denoises, the target model will be largely dependent on the given conditional constraints during the entire denoising process. This approach is not conducive to the autonomous learning of the target model and will result in low denoising ability of the target model without conditional constraints. Based on this, the embodiment of the present disclosure provides a denoising processing scheme based on a conditional shielding strategy, which can improve the denoising ability of the trained diffusion model without conditional constraints to a certain extent, thereby improving the accuracy of the trained diffusion model in generating character perspective videos.

[0245] Please refer to Fig.21 In another embodiment, step 1820 specifically includes but is not limited to the following steps 2110-2140: Step 2110, performing splicing processing on the sample guidance information and the sample latent space feature vector to obtain the sample denoising splicing feature; Step 2120: Based on the sample denoising splicing feature, a first denoising process is performed through a denoising network to obtain a first denoising feature; Step 2130: Based on the sample latent space feature vector, perform a second denoising process through a denoising network to obtain a second denoising feature; Step 2140: perform feature weighting on the first denoising feature and the second denoising feature to obtain a sample video frame denoising feature.

[0246] Steps 1910-1930 are described in detail below.

[0247] In steps 2110-2120, the first denoising feature is used to indicate the result of the denoising network performing a denoising operation on the sample latent space feature under the constraint of the sample guidance information.

[0248] In the specific implementation of this embodiment, the specific process of steps 2110-2120 is similar to that of steps 1910-1930, and will not be described in detail to save space.

[0249] In step 2130, the second denoising feature is used to indicate the result of the denoising network performing a denoising operation on the sample latent space feature without the constraint of sample guidance information.

[0250] In the specific implementation of this embodiment, since the denoising network performs denoising under unconditional constraints, that is, the noise added by the diffusion network to the sample latent space feature vector is eliminated one by one, the feature information in the sample latent space feature vector will not be affected by any conditional constraints and will not be changed. The denoising operation only reduces the noise information carried by the sample latent space feature vector without changing the original feature information. Therefore, after the denoising network performs simple denoising on the sample latent space feature vector, the sample latent space feature vector will be restored to the state before denoising, thereby obtaining the second denoising feature. Among them, the second denoising feature is basically consistent with the sample video frame coding feature.

[0251] In step 2140, first, a first weight and a second weight are determined, wherein the sum of the first weight and the second weight is 1, the first weight is used to indicate the importance of conditionally constrained denoising in model training, and the second weight is used to indicate the importance of unconditionally constrained denoising in model training. Next, the first weight and the first denoising feature are multiplied to obtain a first weighted denoising feature, and the second weight and the second denoising feature are multiplied to obtain a second weighted denoising feature. Finally, the first weighted denoising feature and the second weighted denoising feature are added to obtain a sample video frame denoising feature.

[0252] It should be noted that this embodiment is illustrated with a predetermined time step of T=1 (i.e., the total number of denoising times is 1). When the total number of denoising times is greater than 1, each denoising process is performed according to steps 2110-2140, and the result of the denoising process is the result of feature weighting of the first denoising feature and the second denoising feature. The denoising object of each denoising process is the denoising process result of the last denoising, and the denoising feature of the sample video frame is the weighted result of the first denoising feature and the second denoising feature generated by the last denoising.

[0253] Furthermore, the denoising process of the denoising network of the embodiment of the present disclosure can be expressed as follows: ; in, Describes the denoising process of the denoising network, which is equivalent to a conditional Gaussian distribution used to denoise the result of the current time step t. And the sample guidance information infers the denoising result of the previous time step t-1 . This conditional Gaussian distribution approximates the posterior distribution. represents Gaussian distribution sampling, is a predefined constant. represents the mean of the conditional Gaussian distribution, represents the variance of the conditional Gaussian distribution, is the default identity matrix. Represents the denoising result of the previous time step t-1 The mean is , the variance is Gaussian distribution.

[0254] Among them, the mean of the conditional Gaussian distribution is It can be expressed as follows: ; in, is a preset constant. It is used to control the proportion of the first denoising feature and the second denoising feature in the feature weighting to balance the influence of conditional constraint information and unconditional constraint information on generating the denoising result of the previous time step t-1. It represents the denoising result of the previous time step calculated by the target model with model parameters θ given the sample guidance information c and time step t. It means that the given sample guidance information c is shielded, and the denoising result of the previous time step is calculated by the target model with model parameters θ according to the given time step t. is the attention weight at time step t, refers to the cumulative average attention weight at time step t, ,and There is a detailed explanation and description below and will not be repeated here.

[0255] The benefit of this embodiment is that it takes into account the conditional shielding with a certain probability introduced in the denoising process for the sample latent space feature vector. Specifically, during each denoising, the sample latent space feature vector is denoised under the conditional constraint of the sample guidance information to obtain the denoising result under the conditional constraint (first denoising feature); at the same time, the sample latent space feature vector is denoised under the unconditional constraint to obtain the denoising result under the unconditional constraint (second denoising feature). Furthermore, during each denoising, the first denoising feature and the second denoising feature are fused by feature weighting, so that after multiple denoising operations combining conditional constraints and unconditional constraints, the sample video frame denoising feature is generated. This method can, to a certain extent, improve the denoising ability of the trained diffusion model under unconditional constraints, thereby improving the accuracy of the trained diffusion model in generating character perspective videos.

[0256] Please refer to Fig. 22 In one embodiment, step 1760 specifically includes but is not limited to the following steps 2210-2240: Step 2210: determining a first loss sub-function based on the sample character perspective video and the expected character perspective video; Step 2220: determining a second loss sub-function based on the sample video frame denoising features corresponding to each of the plurality of sample video frames and a preset benchmark denoising feature; Step 2230: Obtain a first loss function based on the first loss sub-function and the second loss sub-function; Step 2240: Based on the first loss function, train the target model to obtain a diffusion model.

[0257] Steps 2210-2240 are described in detail below.

[0258] In step 2210, the first loss sub-function is used to indicate the overall degree of difference between the video information of the sample character's perspective video and the expected character's perspective video.

[0259] When this embodiment is implemented, the sample character perspective video includes a plurality of character perspective video frames, and the expected character perspective video includes reference video frames corresponding one-to-one to the plurality of character perspective video frames.

[0260] The step 2210 may include but is not limited to the following steps: Determine a first pixel value of a pixel point of each character perspective video frame, and determine a second pixel value of a pixel point of each reference video frame; For each character perspective video frame, determining a pixel difference value based on a first pixel value and a second pixel value of a reference video frame corresponding to the character perspective video frame; A first loss sub-function is determined based on respective pixel differences of a plurality of character perspective video frames.

[0261] The first pixel value is used to indicate the pixel value of a pixel point in the character perspective video frame. The second pixel value is used to indicate the pixel value of a pixel point in the reference video frame. The pixel difference value is used to indicate the difference in the pixel value of a pixel point at the same position in the character perspective video frame and the reference video frame.

[0262] Specifically, for each character perspective video frame, the pixel difference between the pixel point at the same position of the character perspective video frame and the corresponding reference video frame is calculated, and the pixel difference is obtained by subtracting the second pixel value from the first pixel value of the pixel point. Then, the mean square error of the pixel difference values ​​of all pixels is calculated to obtain the pixel mean square error of a single character perspective video frame. Finally, the pixel mean square errors of all character perspective video frames are averaged to obtain the first loss subfunction.

[0263] In step 2220, the reference denoising feature is used to indicate the degree of noise added to the reference video frame corresponding to each sample video frame. In the embodiment of the present disclosure, the reference denoising feature may be a feature that conforms to a standard normal distribution.

[0264] The second loss sub-function is used to indicate the overall difference between the predicted noise and the reference noise of the plurality of sample video frames.

[0265] When this embodiment is implemented specifically, step 2220 may include but is not limited to the following steps: For each sample video frame, a regularization term is calculated based on the sample video frame denoising feature and the reference denoising feature to obtain a regularization term calculation result; Based on the regularization term calculation results of each of the plurality of sample video frames, a second loss sub-function is determined.

[0266] The regularization term calculation result is used to indicate the degree of difference between the predicted noise and the reference noise of a single sample video frame.

[0267] Specifically, first, for each sample video frame, the noise difference between the sample video frame denoising feature and the reference denoising feature is calculated to obtain the noise difference; then, the regularization term is calculated on the noise difference to obtain the regularization term calculation result. Finally, the regularization term calculation results of all sample video frames are averaged to obtain the second loss sub-function.

[0268] Among them, the calculation result of the regularization term of each sample video frame can be expressed as shown in the following formula: ; Among them, loss1 refers to the calculation result of the regularization term corresponding to a single sample video frame (character perspective video frame). refers to the baseline denoising feature at time step t, refers to the denoising features of the sample video frame at time step t, , (.) is the target model with model parameters θ, It refers to the target model with model parameters θ that calculates the sample latent space feature vector based on the sample guidance information c The result of performing t denoising operations is: is the latent space feature vector of the sample at time step t, is the sample boot information, Represents the time step, which is used to indicate the current accumulated noise count.

[0269] Furthermore, the sample latent space feature vector It can be expressed as follows: ; in, ; ; in, is a pre-set real number close to 0. It will gradually decrease as the time step t increases. Baseline denoising features is a randomly generated standard normal distribution sample value, the benchmark denoising feature The tensor shape and sample latent space feature vector The tensor has the same shape as . T is the total number of time steps, and t is any integer between 1 and T. is the attention weight at time step t. Refers to the sample video frame encoding features. refers to the accumulated average attention weight at time step t.

[0270] Based on this, the latent space feature vector of the sample at time step t It can be understood that the sample video frame encoding features and the cumulative average attention weight Multiply by the square root of and then add the baseline denoising feature and 1 minus the cumulative average attention weight It is obtained by multiplying the bisection root of .

[0271] In step 2230, the first loss function is used to indicate the overall degree of difference in pixels and noise between the sample character perspective video generated by the target model and the expected character perspective video.

[0272] When this embodiment is implemented, the first loss sub-function and the second loss sub-function are weighted to obtain the first loss function, wherein the specific process of weighted calculation of the first loss sub-function and the second loss sub-function is similar to the specific process of weighting the first denoising feature and the second denoising feature in the above step 2140. The difference is that the weights of the first loss sub-function and the second loss sub-function in step 2230 can be freely set, and the sum of the weights of the two may not be 1. To save space, it will not be repeated.

[0273] In step 2240, the model parameters of the target model are adjusted with minimizing the first loss function as the training goal, and the above steps 1710-1760 are repeated to implement iterative training of the target model. The model parameters that minimize the first loss function are used as the final model parameters, and the target model with the final model parameters is used as the trained diffusion model.

[0274] The benefit of this embodiment is that it takes into account the pixel value differences and noise differences between the sample character perspective videos generated by the target model in each iterative training round and the expected character perspective videos; based on a supervised learning approach, the first loss function is jointly determined based on the pixel difference between the first pixel value of each pixel point in each character perspective video frame and the second pixel value of the pixel point in the reference video frame, as well as the noise feature differences between the sample video frame denoising features of each character video frame and the reference denoising features. This approach can train the target model by minimizing the pixel value differences and noise differences, which is beneficial to improving the model training effect, and thereby improving the accuracy of the model generating character perspective videos.

[0275] Please refer to Fig.23 In one embodiment, the target model is obtained by adjusting the first model by the following steps: Step 2310: Acquire multiple reference video frames of a reference video; Step 2320: Perform three-dimensional convolution processing on multiple reference video frames to obtain prediction means and prediction standard deviations of the multiple reference video frames; Step 2330: re-parameterize the multiple reference video frames based on the predicted mean and the predicted standard deviation to obtain the video frame hidden state features of the multiple reference video frames; Step 2340: Decode the hidden state features of each of the multiple reference video frames to obtain predicted video frames corresponding to each of the multiple reference video frames; Step 2350: Based on the predicted video frame, the reference video frame, the predicted mean and the predicted standard deviation, adjust the parameters of the first model to obtain a target model.

[0276] Steps 2310-2350 are described in detail below.

[0277] In step 2310, the reference video refers to video data corresponding to certain narrative film and television works used to adjust the first model. The multiple reference video frames are video frames obtained by performing video segmentation on the reference video.

[0278] For example, for a reference video corresponding to a reference film or television work, segment it according to a fixed time step (for example, 30 seconds) to obtain a reference video frame set corresponding to the reference video. The reference video frame set is represented as , ,in, represents the reference video frame in time window t, n is the total number of reference video frames, n is an integer, and t is any integer from 1 to n. refers to the model parameters of the first model. Indicates that the reference video frame set is subject to the model parameters of the first model The set of video frames generated by the influence of .

[0279] In the specific implementation of this embodiment, step 2310 is similar to the above-mentioned step 1710. To save space, it will not be described in detail.

[0280] In step 2320, the prediction mean refers to the expected value of the Gaussian distribution corresponding to each of the multiple reference video frames in the latent vector space. The prediction standard deviation refers to the standard deviation of the Gaussian distribution corresponding to each of the multiple reference video frames in the latent vector space.

[0281] It should be noted that each reference video frame has a corresponding prediction mean and prediction standard deviation. The prediction means and prediction standards of different reference video frames may differ to a certain extent.

[0282] In the specific implementation of this embodiment, first, the encoding network of the first model performs three-dimensional convolution processing on each reference video frame, so that the three-dimensional convolution kernel of the encoding network slides on the temporal dimension, height and width of the reference video frame, captures the spatiotemporal features of the reference video frame, and obtains the video frame convolution features of each reference video frame, wherein each reference video frame feature has multiple video frame convolution features. Next, for each reference video frame, the multiple video frame convolution features of the reference video frame are pixel-averaged to obtain the predicted mean corresponding to the reference video frame. Furthermore, the square of the difference between the convolution features of each video frame of the reference video frame and the predicted mean is calculated to obtain a square value, and the square root is taken after averaging the multiple square values ​​to obtain the predicted standard deviation of the reference video frame.

[0283] Among them, the predicted mean can be expressed as , the prediction standard deviation can be expressed as The tensor shape of the predicted mean and predicted standard deviation can be expressed as 64×64×n, where n refers to the total number of reference video frames.

[0284] In step 2330, the video frame hidden state feature is used to indicate the Gaussian distribution corresponding to the reference video frame in the latent vector space.

[0285] In the specific implementation of this embodiment, for each reference video frame feature, a random variable is sampled from a standard normal distribution with a mean of 0 and a standard deviation of 1; then, the random variable is multiplied by the predicted standard deviation to obtain a product result, and then the product result and the predicted mean are added to obtain a Gaussian distribution that meets the requirements, and this generated Gaussian distribution is determined as the hidden state feature of the video frame corresponding to the reference video frame.

[0286] Among them, the reference video frame The corresponding video frame hidden state features can be expressed as follows: ; in, Refers to the hidden state features of the video frame, is the predicted mean, Refers to the predicted standard deviation. Refers to the standard normal distribution with mean 0 and standard deviation 1.

[0287] In step 2340, the predicted video frame refers to the prediction result of the reference video frame generated after the decoding network of the first model decodes the hidden state features of the video frame.

[0288] In the specific implementation of this embodiment, for each reference video frame, the video frame hidden state features of the reference video frame are input into the decoding network of the first model, and the video frame hidden state features are subjected to inverse three-dimensional convolution processing by the decoding network to obtain a predicted video frame corresponding to the reference video frame. The corresponding predicted video frame can be expressed as .

[0289] In step 2350, the specific implementation of this embodiment may include but is not limited to the following steps: A video frame error function is obtained by performing mean square error calculation based on a plurality of reference video frames and predicted video frames corresponding to the plurality of reference video frames; Logarithmic calculation is performed based on the predicted mean and predicted standard deviation to obtain the log-likelihood function; Based on the video frame error function and the log-likelihood function, the parameters of the first model are adjusted to obtain a target model.

[0290] The video frame error function is used to indicate the overall difference between the predicted video frame and the reference video frame, and the log likelihood function is used to indicate the overall difference between the video frame hidden state features of the reference video frame and the standard normal distribution.

[0291] Specifically, when determining the video frame error function, for each reference video frame, for each pixel point of the reference video frame, the square of the difference between the pixel value of the pixel point in the reference video frame and the pixel value of the pixel point in the predicted video frame is calculated to obtain a pixel square value, and the pixel square values ​​of all pixels of the reference video frame are averaged to obtain a sub-error function, and the sub-error functions of all reference video frames are averaged to obtain a video frame error function. The sub-error function can be expressed as . MSE(.) is the mean squared error function.

[0292] When determining the log-likelihood function, for each reference video frame, the predicted mean and the predicted standard deviation are logarithmically calculated according to the first formula to obtain the logarithmic result of the reference video frame. Then, the logarithmic results of multiple reference video frames are averaged to obtain the log-likelihood function. The first formula can be expressed as follows: ; in, Represents the reference video frame The logarithmic result of Represents the reference video frame The predicted mean of Represents the reference video frame The predicted standard deviation.

[0293] Further, the total loss function is obtained by subtracting the log-likelihood function from the video frame error function, and the parameters of the first model are adjusted according to the total loss function, thereby obtaining the target model. The specific method of adjusting the parameters of the first model based on the video frame error function and the log-likelihood function is similar to the above steps 2230-2240. In order to save space, it will not be repeated.

[0294] The benefit of this embodiment is that it allows the first model to learn the predicted mean through the back propagation algorithm during the training process. and the predicted standard deviation The value of and maintain a certain randomness, the randomness is due to the fact that in the back propagation process, the randomly sampled part (that is, normal (0,1)) is regarded as a constant and the predicted mean and the predicted standard deviation As a model parameter, it is continuously updated so that the first model can optimize the predicted mean through gradient descent while maintaining randomness. and the predicted standard deviation , thereby learning the potential distribution of the video data of the reference video frame, thereby improving the learning ability of the first model. Furthermore, when continuously adjusting the model parameters of the first model, the average mean square error and the log-likelihood results of the hidden state features and the standard normal distribution are combined to improve the adjustment accuracy of the first model, thereby improving the basic model performance of the target model used to train the diffusion model.

[0295] Combine the following Figure 24A-24C The specific process of training the diffusion model in the embodiment of the present disclosure is described by way of example.

[0296] like Fig.24A As shown in FIG. 1 , it is a specific schematic diagram of using the target model to generate a character perspective video frame corresponding to a sample video frame in an iterative training round. Specifically, first, the sample video frame is input into the encoding network, and the encoding network generates the encoding features of the sample video frame. , the sample video frame encoding feature is a hidden state feature. Then, the diffusion network encodes the sample video frame feature Perform forward diffusion and add noise to the sample video frame encoding features t times to obtain the sample latent space feature vector Furthermore, the sample role script information, sample image information and sample background description information are input into the conditional fusion network as conditional constraints. , by the conditional fusion network Output sample guidance information. Then, the denoising network performs reverse diffusion on the sample latent space feature vector. At this time, the denoising network will perform t denoising processes on the sample latent space feature vector according to the sample guidance information to obtain the sample video frame denoising feature vector. Finally, the decoding network decodes the character’s perspective video frame based on the denoising features of the sample video frame.

[0297] like Fig. 24B As shown, when the first model is adjusted, a specific schematic diagram of using the first model to generate a predicted video frame corresponding to a reference video frame in an iteration round. Specifically, first, the sample video frame is input into the encoding network, and the encoding network generates a hidden state feature of the video frame. Then, the decoding network decodes the predicted video frame according to the hidden state feature of the video frame.

[0298] like Fig.24C As shown, it is Fig.24A Specific illustration of the diffusion processing of the diffusion network and the denoising processing of the denoising network of the target model. First, the diffusion network encodes the features of the sample video frame Perform forward diffusion and add noise to the sample video frame encoding features t times to obtain the sample latent space feature vector Furthermore, the sample role script information, sample image information and sample background description information are input into the conditional fusion network as conditional constraints. , by the conditional fusion network Output sample guidance information. Then, the denoising network performs reverse diffusion on the sample latent space feature vector. At this time, the denoising network will perform t denoising processes on the sample latent space feature vector according to the sample guidance information to obtain the sample video frame denoising feature vector. . Among them, the denoising network will switch the switch according to the conditional shielding strategy during the denoising process. When the switch is set to the left, the unconditional constraint processing of the sample latent space feature vector is performed, that is, it is not affected by the sample guidance information, and the sample latent space feature vector is directly denoised. At the same time, when the switch is set to the right, the conditional constraint processing of the sample latent space feature vector is performed, that is, the sample latent space feature vector is denoised according to the sample guidance information.

[0299] The following is a detailed description of the training process of the video script description generation model of the embodiment of the present disclosure.

[0300] Please refer to Fig.25 In one embodiment, the video script description generation model is trained by the following steps: Step 2510: Acquire a plurality of reference video frames arranged in time sequence of a reference video, and reference script description information corresponding to each of the plurality of reference video frames; Step 2520, extracting reference video frame features of each of the multiple reference video frames; Step 2530: Based on the reference video frame features of each of the multiple reference video frames and the reference script description information, a script is generated by using a preset second model to obtain predicted script information of each of the multiple reference video frames; Step 2540: Based on the reference script description information and predicted script information of each of the multiple reference video frames, train the second model to obtain a video script description generation model.

[0301] Steps 2510-2540 are described in detail below.

[0302] In step 2510, the reference script description information is used to indicate the script content corresponding to each reference video frame obtained by manual annotation.

[0303] In the specific implementation of this embodiment, the step 2510 of obtaining a plurality of reference video frames arranged in time sequence of the reference video is similar to the above-mentioned step 2310. To save space, it will not be described in detail.

[0304] Furthermore, based on the manual annotation method, the plot corresponding to each reference video frame in the reference video frame set is described and generated to obtain the reference script description information corresponding to each reference video frame. The corresponding reference script description set can be expressed as ,in, is the reference video frame The corresponding reference script description information.

[0305] In step 2520, the reference video frame feature is used to indicate video feature information contained in a single reference video frame.

[0306] In the specific implementation of this embodiment, the reference video frame features are extracted by the feature extraction network of the second model, and the feature extraction process is similar to the above steps 710-740. To save space, it will not be repeated.

[0307] In step 2530, the second model refers to a neural network model that can generate corresponding script description information for the input video frame, wherein the second model mainly includes a feature extraction network based on a convolutional neural network structure, and a script description generation network based on a transformer decoder.

[0308] The predicted script information is used to indicate the script content generated by the second model for a single reference video frame.

[0309] To save space, the specific process of generating a script through a preset second model based on the reference video frame features of each of the multiple reference video frames and the reference script description information in the embodiment of the present disclosure will be described in detail below. No further description will be given here.

[0310] In step 2540, based on the reference script description information and predicted script information of each of the multiple reference video frames, the specific process of training the second model may include but is not limited to the following steps: For each reference video frame, based on the reference video frame script description information and the predicted script description information, determine a third loss sub-function; Determining a second loss function based on the plurality of third loss sub-functions; Based on the second loss function, the second model is trained to obtain a video script description generation model.

[0311] The third loss sub-function is used to indicate the degree of difference between the predicted script description information of a single reference video frame and the script description information of the reference video frame. The second loss function is used for the overall degree of difference between the predicted script description information predicted by the second model for multiple reference video frames and the respective reference video frame script description information.

[0312] Specifically, the predicted script description information of each reference video frame includes the first probability and the first serial number of each of the multiple predicted words, and the reference video frame script description information includes the second serial number of each of the multiple reference words.

[0313] Reference words refer to the individual words in the reference video frame script description information.

[0314] The predicted word is the prediction result of the second model for each element of the reference video frame feature, wherein the predicted word is often a word taken from a preset word list of the second model.

[0315] The first probability is used to indicate the probability of predicting the element of the reference video frame feature as the predicted word.

[0316] The first sequence number is used to indicate the position order of the predicted word in the predicted script description information.

[0317] The second serial number is used to indicate the position order of the reference word in the reference video frame script description information.

[0318] In this embodiment, for each reference video frame, the specific process of determining the third loss sub-function based on the reference video frame script description information and the predicted script description information may include but is not limited to the following steps: For a single predicted word, if the predicted word is consistent with a reference word whose second serial number is the same as the first serial number of the single predicted word, the predicted word is determined as a target word; Based on the negative logarithm of the first probability of the target word, a third loss sub-function is determined.

[0319] Specifically, for each reference video frame, according to the position order of each predicted word in the predicted script description information, the predicted word with the first sequence number 1 is examined to determine whether the predicted word with the first sequence number 1 is the same as the reference word with the second sequence number 1. Only when the predicted word with the first sequence number 1 is the same as the reference word with the second sequence number 1, the predicted word with the first sequence number 1 is used as the target word, and so on, the predicted word with the first sequence number 2 is examined, ..., until the predicted words with the first sequence number equal to the total number of predicted words (the total number of reference words) are examined, and multiple target words are obtained. Further, for each target word, the negative logarithm of the first probability of the target word is taken to obtain the negative logarithm of the first probability, and the negative logarithms of the first probabilities of all target words are added to obtain the third loss sub-function of the reference video frame.

[0320] Among them, the third loss sub-function can be expressed as follows: ; in, Refers to the reference video frame The corresponding third loss sub-function is, Refers to the unique hot vector of the i-th position of the reference script description information. When the predicted word at the i-th position is consistent with the reference word, The value of is 1; when the predicted word at the i-th position is inconsistent with the reference word, The value of is 0. Refers to the probability value corresponding to each predicted word. The probability value of the predicted word at the i-th position is equal to the reference word only when the predicted word at the i-th position is consistent with the reference word. It is the first probability. Indicates the total number of reference words that refer to the script description information. (.) represents the cross entropy loss function.

[0321] Furthermore, the plurality of third loss sub-functions are averaged to obtain a second loss function. Finally, based on the second loss function, the second model is trained to obtain a video script description generation model, wherein the specific method of training the second model based on the second loss function is similar to the above step 2240. To save space, it will not be described in detail.

[0322] The advantage of this embodiment is that, by using a supervised training method, the second model is used to generate a corresponding predicted script information for each reference video frame, and the model parameters of the second model are iteratively adjusted with the training goal of minimizing the difference between the predicted script information and the reference script description information of each reference video frame. This method can improve the accuracy of the trained video script description generation model in generating script information based on video frame features.

[0323] Please refer to Fig.26 In one embodiment, step 2530 specifically includes but is not limited to the following steps 2610-2640: Step 2610: For each reference video frame, embed the reference script description information to obtain the reference script description embedding feature; Step 2620: Based on the reference script description embedded features, self-attention calculation is performed through the second model to obtain a first attention calculation result; Step 2630: Based on the first attention calculation result and the reference video frame feature, a cross attention calculation is performed by a second model to obtain a second attention calculation result; Step 2640, generate a script description based on the second attention calculation result and the preset vocabulary in the second model to obtain the predicted script information of the reference video frame.

[0324] Steps 2610-2640 are described in detail below.

[0325] In step 2610, the reference script description embedding feature is used to indicate the potential representation of the reference script description information in the latent vector space.

[0326] When this embodiment is implemented, the specific process of step 2610 is similar to the above step 1410. To save space, it will not be described again.

[0327] In step 2620, the specific process of performing self-attention calculation on the reference script description embedding feature by the second model is similar to the specific process of performing self-attention calculation on the target video frame feature by using the attention layer in step 620. To save space, it will not be repeated.

[0328] In step 2630, first, the first attention calculation result is linearly projected, and the query channel of the attention layer is used to project the first attention calculation result into a query feature, the key channel of the attention layer is used to project the reference video frame feature into a key feature, and the value channel of the attention layer is used to project the reference video frame feature into a value feature. Next, the query feature and the transposed result of the key feature are inner-producted to obtain an inner-product result, and the inner-product result is divided by the square root of the feature dimension of the key feature to obtain an attention score. Further, the softmax function is used to convert the attention score into an attention weight, and the attention weight and the value feature are multiplied to obtain a second attention calculation result.

[0329] In step 2640, the preset word list is a list consisting of multiple reference words that is preset according to business requirements.

[0330] In the specific implementation of this embodiment, the specific process of generating the script description based on the second attention calculation result and the preset word list in the second model is basically the same as the feedforward processing of step 620 and the process of calculating the probability size of the reference word of each element in the second attention calculation result. To save space, it will not be repeated.

[0331] The benefit of this embodiment is that the self-attention mechanism and the cross-attention mechanism are introduced, so that the second model can identify and emphasize the key features of the reference script description information of the reference video frame. At the same time, the reference video frame features corresponding to the reference video frame are used as key features and value features, so that the second model can pay more attention to the video content related to the current plot when generating the script description, thereby improving the relevance and accuracy of the generated predicted script information. In addition, the second model relies on the transform Decoder structure when generating the script description, which makes the script description generation process more flexible and can adapt to different video content and plot description requirements. Through iterative training, the second model can fully learn how to generate script description information that meets the requirements based on video frame features.

[0332] The following describes the apparatus and devices of the embodiments of the present disclosure.

[0333] It is to be understood that, although the steps in the above-mentioned flowcharts are sequentially displayed according to the characterization of arrows, these steps are not necessarily executed in sequence according to the order of arrow characterization. Unless there is a clear description in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0334] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present application is obtained.

[0335] Fig. 27 The structure diagram of the character perspective video generation device 2700 provided in the embodiment of the present disclosure. The character perspective video generation device 2700 includes: A first acquisition unit 2710 is used to acquire a source video having a target character and source background description information of the source video; An extraction unit 2720, configured to extract source script information from a source video; The second acquisition unit 2730 is used to acquire the introductory language template information corresponding to the role script conversion model; The first generating unit 2740 is used to generate role script information from the perspective of the target role by using the role script conversion model based on the source script information, the source background description information and the guide template information; The second generating unit 2750 is used to generate diffusion guidance information based on the role script information, the picture information corresponding to the target role, and the source background description information, and generate a role perspective video from the perspective of the target role based on the diffusion guidance information.

[0336] Optionally, the first generating unit 2740 includes: An acquisition module (not shown), used to acquire image information corresponding to a target character; The first generation module (not shown) is used to generate introductory words based on source script information, picture information, and source background description information using introductory words template information, and input the introductory words into the role script conversion model to obtain role script information.

[0337] Optionally, the source script information includes source line information and source scene information, the role script information includes role script line information and role script scene information, the introductory language template information includes first introductory language template information and second introductory language template information, and the introductory language includes first introductory language and second introductory language; The first generation module (not shown) is used to: Based on the source dialogue information, source scene information, picture information and source background description information, the first guidance words are generated using the first guidance words template information, and the first guidance words are input into the role script conversion model to obtain the role script dialogue information; Based on the source scene information and the source background description information, the second guide words are generated using the second guide word template information, and the second guide words are input into the role-script conversion model to obtain the role-script scene information.

[0338] Optionally, the role script conversion model includes a first sub-model and a second sub-model; The first generating unit 2740 is used for: Based on the guide template information, the source script information, and the source background description information, an instance is generated through the first sub-model to obtain a role script prompt instance; Based on the role script prompt instance, the script is generated through the second sub-model to obtain the role script information.

[0339] Optionally, the second acquiring unit 2730 includes: A segmentation module (not shown), configured to segment the source video to obtain a plurality of source video frames arranged in time sequence; A second generation module (not shown) is used to input a plurality of source video frames into a pre-trained video script description generation model for description generation, and obtain video frame script description information of each of the plurality of source video frames; An integration module (not shown) is used to integrate the video frame script description information of each of the multiple source video frames to obtain source script information.

[0340] Optionally, the video script description generation model includes a feature extraction network and a script description generation network; The second generation module (not shown) includes: An extraction submodule (not shown) is used to extract features of each source video frame based on a feature extraction network to obtain target video frame features of the source video frame; A generation submodule (not shown) is used to generate a script based on the target video frame features using a script description generation network to obtain video frame script description information of the source video frame.

[0341] Optionally, an extraction submodule (not shown) is used to: For each source video frame, multi-scale downsampling is performed on the source video frame based on a feature extraction network to obtain a first downsampling feature, a second downsampling feature, and a third downsampling feature, wherein the feature dimensions of the first downsampling feature, the second downsampling feature, and the third downsampling feature are increased in sequence; Performing dimensionality reduction processing on the first down-sampling feature, the second down-sampling feature, and the third down-sampling feature, respectively, to obtain a first dimensionality reduction feature corresponding to the first down-sampling feature, a second dimensionality reduction feature corresponding to the second down-sampling feature, and a third dimensionality reduction feature corresponding to the third down-sampling feature; The first dimensionality reduction feature, the second dimensionality reduction feature, and the third dimensionality reduction feature are concatenated to obtain an initial video frame feature of the source video frame; The initial video frame features of the source video frame and the initial video frame features of the previous source video frame of the source video frame are spliced ​​to obtain the target video frame features of the source video frame.

[0342] Optionally, an integration module (not shown) is used to: splicing the video frame script description information of each of the multiple source video frames to obtain preliminary script information; The preliminary script information is polished to obtain the source script information.

[0343] Optionally, the second generating unit 2750 is used to: Encoding the source background description information to obtain background description encoding features; Encoding the role script information to obtain the role script encoding features; Extract features from the image information to obtain the image features of the target character; Based on the background description coding features, the role script coding features, and the target role image features, conditional information is generated through a preset conditional fusion network to obtain diffusion guidance information.

[0344] Optionally, the second generating unit 2750 is used to: guide the diffusion model based on the diffusion guiding information to generate a character perspective video of the target character perspective; The diffusion model is trained in the following way: A sample acquisition unit (not shown), configured to acquire a sample video having a sample character, sample background description information of the sample video, and an expected character perspective video of the sample character; A sample extraction unit (not shown), used to extract sample script information from a sample video; A sample script generating unit (not shown) is used to generate sample role script information from the perspective of a sample role using a role script conversion model based on the sample script information, the sample background description information and the preset introductory language template information; A sample guide information generating unit (not shown) is used to generate sample guide information based on sample role script information, sample picture information corresponding to the sample role, and sample background description information; A sample video generating unit (not shown) is used to generate a video through a target model based on the sample guidance information to obtain a sample character perspective video of the sample character perspective; The first training unit (not shown) is used to train the target model based on the sample character perspective video and the expected character perspective video to obtain a diffusion model.

[0345] Optionally, the sample video includes a plurality of sample video frames; the target model includes a diffusion network, a denoising network, and a decoding network; The sample video generation unit (not shown) is used to: A diffusion module (not shown) is used to perform diffusion processing on the sample video frame encoding features of the sample video frame based on the diffusion network for each sample video frame to obtain a sample latent space feature vector; A denoising module (not shown), configured to perform denoising processing on the sample latent space feature vector through a denoising network based on the sample guidance information to obtain a denoising feature of the sample video frame; A decoding module (not shown), configured to decode the denoising features of the sample video frames based on a decoding network to obtain predicted video frames of the sample video frames; The video generation module (not shown) is used to obtain a sample character perspective video based on the predicted video frames of each of the multiple sample video frames.

[0346] Optionally, the denoising network includes an upsampling module and a downsampling module; The denoising module (not shown) is used to: The sample guidance information and the sample latent space feature vector are spliced ​​to obtain the sample denoising splicing feature; Based on the downsampling module, the sample denoising splicing features are downsampled to obtain the video frame downsampling features; The down-sampled features of the video frames are up-sampled based on the up-sampling module to obtain the denoising features of the sample video frames.

[0347] Optionally, a denoising module (not shown) is used to: The sample guidance information and the sample latent space feature vector are spliced ​​to obtain the sample denoising splicing feature; Based on the sample denoising splicing feature, a first denoising process is performed through a denoising network to obtain a first denoising feature; Based on the sample latent space feature vector, a second denoising process is performed through a denoising network to obtain a second denoising feature; The first denoising feature and the second denoising feature are weighted to obtain a sample video frame denoising feature.

[0348] Optionally, the sample video includes a plurality of sample video frames; The first training unit (not shown) is used to: Determining a first loss sub-function based on the sample character perspective video and the expected character perspective video; Determining a second loss sub-function based on sample video frame denoising features corresponding to each of the plurality of sample video frames and a preset benchmark denoising feature; Based on the first loss sub-function and the second loss sub-function, a first loss function is obtained; Based on the first loss function, the target model is trained to obtain the diffusion model.

[0349] Optionally, determining the second loss sub-function based on sample video frame denoising features corresponding to each of the plurality of sample video frames and a preset reference denoising feature includes: For each sample video frame, a regularization term is calculated based on the sample video frame denoising feature and the reference denoising feature to obtain a regularization term calculation result; Based on the regularization term calculation results of each of the plurality of sample video frames, a second loss sub-function is determined.

[0350] Optionally, the sample character perspective video includes a plurality of character perspective video frames, and the expected character perspective video includes reference video frames corresponding one-to-one to the plurality of character perspective video frames; Based on the sample character perspective video and the expected character perspective video, a first loss sub-function is determined, including: Determine a first pixel value of a pixel point of each character perspective video frame, and determine a second pixel value of a pixel point of each reference video frame; For each character perspective video frame, determining a pixel difference value based on a first pixel value and a second pixel value of a reference video frame corresponding to the character perspective video frame; A first loss sub-function is determined based on respective pixel differences of a plurality of character perspective video frames.

[0351] Optionally, the target model is obtained by adjusting the first model through the following steps: A reference video frame acquisition unit (not shown), configured to acquire a plurality of reference video frames of a reference video; A convolution unit (not shown), configured to perform three-dimensional convolution processing on the multiple reference video frames to obtain predicted means and predicted standard deviations of the multiple reference video frames; A reparameterization unit (not shown) is used to reparameterize the multiple reference video frames respectively based on the predicted mean and the predicted standard deviation to obtain the video frame hidden state features of the multiple reference video frames; A feature decoding unit (not shown) is used to decode the hidden state features of each of the multiple reference video frames to obtain predicted video frames corresponding to each of the multiple reference video frames; An adjustment unit (not shown) is used to adjust parameters of the first model based on the predicted video frame, the reference video frame, the predicted mean and the predicted standard deviation to obtain a target model.

[0352] Optionally, the adjustment unit (not shown) is used to: A video frame error function is obtained by performing mean square error calculation based on a plurality of reference video frames and predicted video frames corresponding to the plurality of reference video frames; Logarithmic calculation is performed based on the predicted mean and predicted standard deviation to obtain the log-likelihood function; Based on the video frame error function and the log-likelihood function, the parameters of the first model are adjusted to obtain a target model.

[0353] Optionally, the video script description generation model is trained by: A reference video acquisition unit (not shown), configured to acquire a plurality of reference video frames arranged in time sequence of a reference video, and reference script description information corresponding to each of the plurality of reference video frames; a feature extraction unit (not shown), configured to extract reference video frame features of each of the plurality of reference video frames; A script generation unit (not shown) is used to generate a script through a preset second model based on the reference video frame features of each of the multiple reference video frames and the reference script description information, so as to obtain predicted script information of each of the multiple reference video frames; The second training unit (not shown) is used to train a second model based on the reference script description information and predicted script information of each of the multiple reference video frames to obtain a video script description generation model.

[0354] Optionally, the script generation unit (not shown) is used to: For each reference video frame, embedding the reference script description information to obtain the reference script description embedding features; Based on the embedded features of the reference script description, the second model performs self-attention calculation to obtain the first attention calculation result; Based on the first attention calculation result and the reference video frame feature, a cross attention calculation is performed through a second model to obtain a second attention calculation result; Based on the second attention calculation result and the preset vocabulary in the second model, the script description is generated to obtain the predicted script information of the reference video frame.

[0355] Optionally, the second training unit (not shown) is used to: For each reference video frame, based on the reference video frame script description information and the predicted script description information, determine a third loss sub-function; Determining a second loss function based on the plurality of third loss sub-functions; Based on the second loss function, the second model is trained to obtain a video script description generation model.

[0356] Optionally, the predicted script description information includes first probabilities and first serial numbers of the plurality of predicted words, and the reference video frame script description information includes second serial numbers of the plurality of reference words; For each reference video frame, based on the reference video frame script description information and the predicted script description information, a third loss sub-function is determined, including: For a single predicted word, if the predicted word is consistent with a reference word whose second serial number is the same as the first serial number of the single predicted word, the predicted word is determined as a target word; Based on the negative logarithm of the first probability of the target word, a third loss sub-function is determined.

[0357] Reference Fig.28 , Fig.28 A structural block diagram of a terminal for implementing the method for generating a character perspective video according to an embodiment of the present disclosure. The terminal may be Figure 1 The terminal includes: a radio frequency (RF) circuit 2810, a memory 2815, an input unit 2830, a display unit 2840, a sensor 2850, an audio circuit 2860, a wireless fidelity (WiFi) module 2870, a processor 2880, and a power supply 2890. Those skilled in the art will understand that Fig.28 The terminal structure shown does not constitute a limitation on the mobile phone or computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0358] The RF circuit 2810 may be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to the processor 2880 for processing; in addition, the designed uplink data is sent to the base station.

[0359] The memory 2815 may be used to store software programs and modules. The processor 2880 executes various functional applications and data processing of the target terminal by running the software programs and modules stored in the memory 2815 .

[0360] The input unit 2830 may be used to receive input digital or character information and generate key signal input related to the setting and function control of the target terminal. Specifically, the input unit 2830 may include a touch panel 2831 and other input devices 2832.

[0361] The display unit 2840 may be used to display input information or provided information and various menus of the target terminal. The display unit 2840 may include a display panel 2841.

[0362] The audio circuit 2860, the speaker 2861, and the microphone 2862 may provide an audio interface.

[0363] In this embodiment, the processor 2880 included in the terminal can execute the character perspective video generation method of the previous embodiment.

[0364] Fig.29 FIG. 1 is a block diagram of a portion of a server for implementing the method for generating a character perspective video according to an embodiment of the present disclosure. The server may be Figure 1 The video processing server shown. The server may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 2922 (for example, one or more processors) and memory 2932, one or more storage media 2930 (for example, one or more mass storage devices) storing application programs 2942 or data 2944. Among them, the memory 2932 and the storage medium 2930 can be short-term storage or permanent storage. The program stored in the storage medium 2930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 2922 can be configured to communicate with the storage medium 2930 and execute a series of instruction operations in the storage medium 2930 on the server.

[0365] The server may also include one or more power supplies 2926, one or more wired or wireless network interfaces 2950, ​​one or more input and output interfaces 2958, and / or one or more operating systems 2941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0366] The central processor 2922 in the server can be used to execute the character perspective video generation method of the embodiment of the present disclosure.

[0367] The embodiments of the present disclosure also provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the character perspective video generation method of each of the aforementioned embodiments.

[0368] The embodiment of the present disclosure also provides a computer program product, which includes a computer program. A processor of an electronic device reads and executes the computer program, so that the electronic device implements the above-mentioned method for generating a character perspective video.

[0369] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0370] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0371] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood as not including the number, and above, below, within, etc. are understood as including the number.

[0372] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0373] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0374] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0375] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0376] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects. The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for generating a character perspective video, characterized in that: The method comprises: Acquire a source video having a target character and source background description information of the source video; Extracting source script information from the source video; Obtain the introductory template information corresponding to the role script conversion model; Based on the source script information, the source background description information and the guide template information, the role script information from the perspective of the target character is generated by using the role script conversion model; Based on the role script information, the picture information corresponding to the target role, and the source background description information, diffusion guidance information is generated, and based on the diffusion guidance information, a role perspective video from the perspective of the target role is generated.

2. The method for generating a character perspective video according to claim 1, characterized in that: The step of generating the role script information from the perspective of the target role by using the role script conversion model based on the source script information, the source background description information and the guide information includes: Obtaining picture information corresponding to the target character; Based on the source script information, the picture information, and the source background description information, the guide words are generated using the guide word template information, and the guide words are input into the role script conversion model to obtain the role script information.

3. The method for generating a character perspective video according to claim 2, characterized in that: The source script information includes source line information and source scene information, the role script information includes role script line information and role script scene information, the guide language template information includes first guide language template information and second guide language template information, and the guide language includes first guide language and second guide language; The method of generating a guide word based on the source script information, the picture information, and the source background description information using the guide word template information, and inputting the guide word into the role script conversion model to obtain the role script information includes: Based on the source dialogue information, the source scene information, the picture information and the source background description information, the first guide language template information is used to generate a first guide language, and the first guide language is input into the role script conversion model to obtain the role script dialogue information; Based on the source scene information and the source background description information, the second guide language is generated using the second guide language template information, and the second guide language is input into the role-script conversion model to obtain the role-script scene information.

4. The method for generating a character perspective video according to claim 1, characterized in that: The extracting source script information from the source video includes: Segmenting the source video to obtain a plurality of source video frames arranged in time sequence; Inputting the plurality of source video frames into a pre-trained video script description generation model for description generation, and obtaining video frame script description information of each of the plurality of source video frames; The video frame script description information of each of the multiple source video frames is integrated to obtain the source script information.

5. The method for generating a character perspective video according to claim 4, characterized in that: The video script description generation model includes a feature extraction network and a script description generation network; The step of inputting the plurality of source video frames into a pre-trained video script description generation model for description generation to obtain video frame script description information of each of the plurality of source video frames includes: For each of the source video frames, extract features of the source video frame based on the feature extraction network to obtain target video frame features of the source video frame; Based on the target video frame features, the script description generation network is used to generate a script to obtain the video frame script description information of the source video frame.

6. The method for generating a character perspective video according to claim 5, characterized in that: The step of extracting features of each of the source video frames based on the feature extraction network to obtain target video frame features of the source video frames includes: For each of the source video frames, performing multi-scale downsampling on the source video frame based on the feature extraction network to obtain a first downsampling feature, a second downsampling feature, and a third downsampling feature, wherein the feature dimensions of the first downsampling feature, the second downsampling feature, and the third downsampling feature are increased in sequence; Performing dimensionality reduction processing on the first down-sampling feature, the second down-sampling feature, and the third down-sampling feature, respectively, to obtain a first dimensionality reduction feature corresponding to the first down-sampling feature, a second dimensionality reduction feature corresponding to the second down-sampling feature, and a third dimensionality reduction feature corresponding to the third down-sampling feature; Performing splicing processing on the first dimensionality reduction feature, the second dimensionality reduction feature, and the third dimensionality reduction feature to obtain an initial video frame feature of the source video frame; The initial video frame features of the source video frame and the initial video frame features of a previous source video frame of the source video frame are spliced ​​to obtain target video frame features of the source video frame.

7. The method for generating a character perspective video according to claim 1, characterized in that: The step of generating diffusion guidance information based on the role script information, the picture information corresponding to the target role, and the source background description information includes: Encoding the source background description information to obtain background description encoding features; Encoding the role script information to obtain role script encoding features; Extracting features from the image information to obtain features of the target character image; Based on the background description coding features, the role script coding features, and the target role image features, conditional information is generated through a preset conditional fusion network to obtain the diffusion guidance information.

8. The method for generating a character perspective video according to claim 4, characterized in that: The step of integrating the video frame script description information of each of the plurality of source video frames to obtain the source script information includes: splicing the video frame script description information of each of the multiple source video frames to obtain preliminary script information; The preliminary script information is polished to obtain the source script information.

9. The method for generating a character perspective video according to claim 1, characterized in that: The generating the character perspective video of the target character perspective based on the diffusion guidance information includes: guiding a diffusion model based on the diffusion guidance information to generate the character perspective video of the target character perspective; The diffusion model is trained in the following way: Acquire a sample video having a sample character, sample background description information of the sample video, and an expected character perspective video of the sample character; Extracting sample script information from the sample video; Based on the sample script information, the sample background description information and the preset introductory language template information, using the role script conversion model, generate the sample role script information from the perspective of the sample role; Generate sample guidance information based on the sample role script information, the sample picture information corresponding to the sample role, and the sample background description information; Based on the sample guidance information, a video is generated through a target model to obtain a sample character perspective video of the sample character perspective; The target model is trained based on the sample character perspective video and the expected character perspective video to obtain the diffusion model.

10. The method for generating a character perspective video according to claim 9, characterized in that: The sample video includes a plurality of sample video frames; the target model includes a diffusion network, a denoising network, and a decoding network; The step of generating a video based on the sample guide information by using a target model to obtain a sample role perspective video of the sample role perspective includes: For each of the sample video frames, diffusion processing is performed on the sample video frame encoding features of the sample video frame based on the diffusion network to obtain a sample latent space feature vector; Based on the sample guidance information, the sample latent space feature vector is denoised by the denoising network to obtain a sample video frame denoising feature; Decoding the denoising features of the sample video frames based on the decoding network to obtain predicted video frames of each of the sample video frames; The sample character perspective video is obtained based on the predicted video frames of each of the multiple sample video frames.

11. The method for generating a character perspective video according to claim 10, characterized in that: The denoising network includes an upsampling module and a downsampling module; The denoising process is performed on the sample latent space feature vector based on the sample guide information by the denoising network to obtain the sample video frame denoising feature, including: Performing splicing processing on the sample guide information and the sample latent space feature vector to obtain a sample denoising splicing feature; Downsampling the sample denoising splicing features based on the downsampling module to obtain video frame downsampling features; The down-sampled features of the video frame are up-sampled based on the up-sampling module to obtain the denoising features of the sample video frame.

12. The method for generating a character perspective video according to claim 10, characterized in that: The denoising process is performed on the sample latent space feature vector based on the sample guide information by the denoising network to obtain the sample video frame denoising feature, including: Performing splicing processing on the sample guide information and the sample latent space feature vector to obtain a sample denoising splicing feature; Based on the sample denoising splicing features, performing a first denoising process through the denoising network to obtain a first denoising feature; Based on the sample latent space feature vector, performing a second denoising process through the denoising network to obtain a second denoising feature; The first denoising feature and the second denoising feature are weighted to obtain the sample video frame denoising feature.

13. The method for generating a character perspective video according to claim 10, characterized in that: The sample video includes a plurality of sample video frames; The step of training the target model based on the sample role perspective video and the expected role perspective video to obtain the diffusion model includes: Determining a first loss sub-function based on the sample character perspective video and the expected character perspective video; Determining a second loss sub-function based on the sample video frame denoising features corresponding to each of the plurality of sample video frames and a preset benchmark denoising feature; Based on the first loss sub-function and the second loss sub-function, a first loss function is obtained; Based on the first loss function, the target model is trained to obtain the diffusion model.

14. The method for generating a character perspective video according to claim 9, characterized in that: The target model is obtained by adjusting the first model through the following steps: Acquire multiple reference video frames of a reference video; Performing three-dimensional convolution processing on the multiple reference video frames to obtain predicted means and predicted standard deviations of the multiple reference video frames; Based on the predicted mean and the predicted standard deviation, the multiple reference video frames are respectively re-parameterized to obtain video frame hidden state features of the multiple reference video frames; Decoding the video frame hidden state features of each of the multiple reference video frames to obtain predicted video frames corresponding to each of the multiple reference video frames; Based on the predicted video frame, the reference video frame, the predicted mean and the predicted standard deviation, the parameters of the first model are adjusted to obtain the target model.

15. The method for generating a character perspective video according to claim 4, characterized in that: The video script description generation model is trained in the following way: Acquire a plurality of reference video frames arranged in time sequence of a reference video, and reference script description information corresponding to each of the plurality of reference video frames; extracting reference video frame features of each of the multiple reference video frames; Based on the reference video frame features of each of the multiple reference video frames and the reference script description information, script generation is performed through a preset second model to obtain predicted script information of each of the multiple reference video frames; Based on the reference script description information and the predicted script information of each of the multiple reference video frames, the second model is trained to obtain the video script description generation model.

16. The method for generating a character perspective video according to claim 15, characterized in that: The step of generating a script based on the reference video frame features of each of the plurality of reference video frames and the reference script description information by using a preset second model to obtain predicted script information of each of the plurality of reference video frames includes: For each of the reference video frames, embedding the reference script description information to obtain a reference script description embedding feature; Based on the reference script description embedding feature, performing self-attention calculation through the second model to obtain a first attention calculation result; Based on the first attention calculation result and the reference video frame feature, performing cross attention calculation through the second model to obtain a second attention calculation result; Based on the second attention calculation result and the preset vocabulary in the second model, a script description is generated to obtain the predicted script information of the reference video frame.

17. A device for generating a character perspective video, characterized in that: The device comprises: A first acquisition unit, configured to acquire a source video having a target character and source background description information of the source video; An extraction unit, used for extracting source script information from the source video; The second acquisition unit is used to acquire the introductory language template information corresponding to the role script conversion model; A first generating unit, configured to generate role script information from the perspective of the target role using the role script conversion model based on the source script information, the source background description information and the guide template information; The second generating unit is used to generate diffusion guidance information based on the role script information, the picture information corresponding to the target role, and the source background description information, and generate a role perspective video from the perspective of the target role based on the diffusion guidance information.

18. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it implements the character perspective video generation method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a character perspective video according to any one of claims 1 to 16 is implemented.

20. A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of an electronic device, so that the electronic device executes the method for generating a character perspective video according to any one of claims 1 to 16.

Images