Video generation method and device, equipment, medium and product

Abstract

The invention discloses a video generation method and device, equipment, a medium and a product, and the method comprises the steps: obtaining a first video, so as to enable each frame of image in the first video to comprise a target object; then, for any frame of image in the first video, performing body shape parameter prediction processing on the image to obtain a body shape parameter prediction result corresponding to the image, and according to body shape adjustment information specified for the target object, performing adjustment processing on the body shape parameter prediction result corresponding to the image to obtain a body shape parameter prediction result corresponding to the image; obtaining a body shape parameter adjustment result corresponding to the image; and finally, according to the first video and the body shape parameter adjustment result corresponding to each frame of image in the first video, generating a second video, so that the second video is used for representing the body shape adjustment result for the first video, thereby facilitating improvement of the body shape adjustment effect.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video generation method, apparatus, device, medium, or product. Background Technology

[0002] For some application scenarios, such as video shooting, video enhancement, or video modification, there is a need to adjust the body shape of objects presented in a video, such as people or virtual characters, such as making them thinner or fatter. Summary of the Invention

[0003] This application provides a video generation method, apparatus, device, medium, and product that helps improve the effect of body shape adjustment.

[0004] To achieve the above objectives, the technical solution provided in this application is as follows:

[0005] This application provides a video generation method, the method comprising: acquiring a first video, wherein each frame of the first video includes a target object; for any frame of the first video, performing body shape parameter prediction processing on the image to obtain a body shape parameter prediction result corresponding to the image, the body shape parameter prediction result being used to describe the body shape of the target object in the image; for any frame of the first video, adjusting the body shape parameter prediction result corresponding to the image based on body shape adjustment information specified for the target object to obtain a body shape parameter adjustment result corresponding to the image; and based on the first video and each frame of the first video... Based on the corresponding body shape parameter adjustment results, a second video is generated. The i-th frame in the second video represents the body shape adjustment result based on the i-th frame in the first video. The i-th frame in the second video is generated based on multiple frames arranged sequentially in the first video and the corresponding body shape parameter adjustment results. The distance between the arrangement position of each frame in the multiple frames in the first video and the arrangement position of the i-th frame in the first video is not greater than a preset distance threshold. i is a positive integer, and i ≤ the number of image frames in the second video or the number of image frames in the first video.

[0006] In one possible implementation, the body shape described by the i-th frame image in the second video is different from the body shape described by the i-th frame image in the first video, and the other information described by the i-th frame image in the second video, except for the body shape, is consistent with the other information described by the i-th frame image in the first video, except for the body shape.

[0007] In one possible implementation, the predicted body shape parameters for each frame of the second video remain consistent.

[0008] In one possible implementation, the second video is generated using a target model; the target model includes at least one processing unit, the processing unit including an image-level processing module and a temporal module, the image-level processing module being used to implement individual processing procedures for each frame of the multi-frame images, and the temporal module being used to perform attention processing on the execution results of the individual processing procedures for the multi-frame images in the temporal direction.

[0009] In one possible implementation, the at least one processing unit includes a plurality of upsampling modules, a timing module corresponding to each of the upsampling modules, a plurality of downsampling modules, and a timing module corresponding to each of the downsampling modules; for any one of the upsampling modules, the timing module corresponding to the upsampling module is used to perform attention processing on the output data of the upsampling module in the time direction; for any one of the downsampling modules, the timing module corresponding to the downsampling module is used to perform attention processing on the output data of the downsampling module in the time direction.

[0010] In one possible implementation, the attention processing is further implemented based on the temporal information of each frame in the multi-frame images; for any frame in the multi-frame images, the temporal information of the image is used to describe the position of the image in the first video.

[0011] In one possible implementation, the timing module includes an integration network, a first transformation network, at least one self-attention network, a second transformation network, and a splitting network. The integration network integrates the input data of the timing module to obtain an integration result, the size of which differs from the size of the input data of the timing module. The input data of the timing module includes the execution results of individual processing procedures for the multi-frame images. The first transformation network converts the integration result from a first representation to a second representation to obtain a first transformation result, the first representation describing image features and the second representation describing video features. The at least one self-attention network performs self-attention processing on the first transformation result to obtain a self-attention processing result. The second transformation network converts the self-attention processing result from the second representation to the first representation to obtain a second transformation result. The splitting network splits the second transformation result to obtain a splitting result, the size of which is the same as the size of the input data of the timing module.

[0012] In one possible implementation, the second video is generated using a target model; the target model includes at least one temporal module; the training process of the target model includes: setting some or all parameters of each temporal module in the target model to zero, and training the other parts of the target model other than the at least one temporal module based on a first image and a second image, wherein the object described in the first image and the object described in the second image are the same object; freezing the parameters of the other parts of the target model other than the at least one temporal module, and training the at least one temporal module in the target model based on a first image sequence and a second image sequence, wherein the object described in the first image sequence and the object described in the second image sequence are the same object.

[0013] In one possible implementation, the timing module includes a second transformation network; setting some or all parameters of each timing module in the target model to zero includes setting the parameters of the second transformation network of each timing module in the target model to zero.

[0014] In one possible implementation, the training process of the at least one temporal module includes: acquiring the first image sequence, the second image sequence, and label information corresponding to each frame image in the second image sequence; performing masking processing on at least one frame image in the first image sequence to obtain a processed sequence, the processed sequence including the masking processing result of the at least one frame image, wherein for any frame image in the at least one frame image, the information described by the masking processing result of that image is less than the information described by that image; determining the prediction information corresponding to each frame image in the second image sequence based on the target model, the processed sequence, and the body shape parameter prediction results corresponding to each frame image in the second image sequence; and updating at least one temporal module in the target model based on the prediction information corresponding to each frame image in the second image sequence and the label information corresponding to each frame image in the second image sequence.

[0015] In one possible implementation, the training process of the at least one temporal module includes: acquiring the label noise corresponding to each frame image in the second image sequence; for any frame image in the second image sequence, adding noise to the image according to the label noise corresponding to that image to obtain the noise-added result corresponding to that image; determining the prediction information corresponding to each frame image in the second image sequence based on the target model, the first image sequence, the body shape parameter prediction results corresponding to each frame image in the second image sequence, and the noise-added results corresponding to each frame image in the second image sequence, the prediction information including prediction noise and prediction image; determining a first loss based on the label noise corresponding to each frame image in the second image sequence and the prediction noise corresponding to each frame image in the second image sequence; determining a second loss based on the region segmentation results of each frame image in the second image sequence, the object position detection results of each frame image in the second image sequence, and the region segmentation results of the prediction image corresponding to each frame image in the second image sequence; and updating at least one temporal module in the target model based on the sum of the first loss and the second loss.

[0016] This application provides a video generation apparatus, comprising: a data acquisition unit for acquiring a first video, wherein each frame of the first video includes a target object; a parameter prediction unit for performing body shape parameter prediction processing on any frame of the first video to obtain a body shape parameter prediction result corresponding to the image, wherein the body shape parameter prediction result is used to describe the body shape of the target object in the image; a parameter adjustment unit for adjusting the body shape parameter prediction result corresponding to any frame of the first video based on body shape adjustment information specified for the target object to obtain a body shape parameter adjustment result corresponding to the image; and a video generation unit for performing data acquisition processing on the first video, wherein each frame of the first video includes a target object; a parameter prediction unit for performing data acquisition processing on any frame of the first video, based on body shape adjustment information specified for the target object; and a parameter generation unit for performing data acquisition processing on any frame of the first video, based on body shape adjustment information specified for the target object; and a parameter adjustment unit for performing data acquisition processing on any frame of the first video, based on body shape parameter prediction processing information specified for the target object; and a parameter generation unit for performing data acquisition processing on any frame of the first video, including a target object; and a parameter adjustment ... adjustment unit for performing data acquisition processing on any frame of the first video, including a target object; and a parameter adjustment unit for performing data acquisition processing on any frame of the first video, including a target object; and a parameter adjustment unit for performing data acquisition processing on A second video is generated from a video and the body shape parameter adjustment results corresponding to each frame of the first video. The i-th frame of the second video is used to represent the body shape adjustment result for the i-th frame of the first video. The i-th frame of the second video is generated based on multiple frames of images arranged sequentially in the first video and the body shape parameter adjustment results corresponding to the multiple frames. The distance between the arrangement position of each frame of the multiple frames in the first video and the arrangement position of the i-th frame of the first video in the first video is not greater than a preset distance threshold. i is a positive integer, and i ≤ the number of image frames in the second video or the number of image frames in the first video.

[0017] This application provides an electronic device, the device comprising: a processor and a memory; the memory for storing instructions or computer programs; the processor for executing the instructions or computer programs in the memory to cause the electronic device to perform the video generation method provided in this application.

[0018] This application provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform the video generation method provided in this application.

[0019] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the video generation method provided in this application.

[0020] Compared with related technologies, this application has at least the following advantages:

[0021] In the technical solution provided in this application, a first video is first acquired so that each frame of the first video includes the target object; secondly, for any frame of the first video, body shape parameter prediction processing is performed on the image to obtain the corresponding body shape parameter prediction result, so that the body shape parameter prediction result can be used to describe the body shape of the target object in the image; then, for any frame of the first video, based on the body shape adjustment information specified for the target object (such as losing 20% ​​of weight), the corresponding body shape parameter prediction result is adjusted to obtain the corresponding body shape parameter adjustment result; finally, based on the first video and the corresponding body shape parameter adjustment results of each frame of the first video, a second video is generated so that the second video can be used to represent the body shape adjustment result for the first video, such that the i-th frame of the second video is used to represent the body shape adjustment result for the i-th frame of the first video, where i is a positive integer and i ≤ the number of image frames in the second video or the number of image frames in the first video.

[0022] In this process, the i-th frame image in the second video is generated based on multiple frames arranged sequentially in the first video and the body shape parameter adjustment results corresponding to those multiple frames. Furthermore, the distance between the arrangement position of each frame image in the first video and the arrangement position of the i-th frame image in the first video is not greater than a preset distance threshold. This ensures that the i-th frame image in the second video not only satisfies the body shape constraints described by the body shape parameter adjustment results corresponding to the i-th frame image in the first video, but also satisfies other constraints described by the multiple frames besides body shape, such as the temporal dependencies of some information. As a result, the second video has better temporal consistency, which is beneficial for improving the body shape adjustment effect. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a video generation method provided in this application embodiment;

[0025] Figure 2 A schematic diagram of a video generation process provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram of a diffusion model provided in an embodiment of this application;

[0027] Figure 4 A schematic diagram of a timing module provided in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] Research has revealed that some body shape adjustment solutions can be implemented as follows: First, body shape adjustment processing is performed on the nth frame of the video to obtain the adjustment result corresponding to the nth frame, where n is a positive integer, n≤N, and N represents the number of frames in the video; then, the adjustment results corresponding to these frames are arranged according to their positions in the video to obtain the adjustment result corresponding to the video, so that the adjustment result corresponding to the video includes the adjustment results corresponding to these frames.

[0031] The study also found that the above-described scheme has the following drawbacks: because the scheme is obtained by adjusting the body shape of each frame of the video separately, the adjustment result of the video is prone to some abrupt changes in time, such as background jumps between two adjacent frames, large changes in body shape between two adjacent frames, and large changes in posture between two adjacent frames. As a result, the adjustment result has poor temporal consistency, which affects the body shape adjustment effect.

[0032] Based on the above research, in order to better improve the body shape adjustment effect, this application provides a video generation method, which includes: firstly, acquiring a first video, such that each frame of the first video includes a target object; secondly, for any frame of the first video, performing body shape parameter prediction processing on the image to obtain the corresponding body shape parameter prediction result, so that the body shape parameter prediction result can be used to describe the body shape of the target object in the image; then, for any frame of the first video, adjusting the corresponding body shape parameter prediction result based on body shape adjustment information specified for the target object (such as losing 20% ​​of weight), to obtain the corresponding body shape parameter adjustment result; finally, generating a second video based on the first video and the corresponding body shape parameter adjustment results of each frame of the first video, so that the second video can be used to represent the body shape adjustment result for the first video, such that the i-th frame of the second video is used to represent the body shape adjustment result for the i-th frame of the first video, where i is a positive integer, and i ≤ the number of frames in the second video or the number of frames in the first video.

[0033] In this process, the i-th frame image in the second video is generated based on multiple frames arranged sequentially in the first video and the body shape parameter adjustment results corresponding to those multiple frames. Furthermore, the distance between the arrangement position of each frame image in the first video and the arrangement position of the i-th frame image in the first video is not greater than a preset distance threshold. This ensures that the i-th frame image in the second video not only satisfies the body shape constraints described by the body shape parameter adjustment results corresponding to the i-th frame image in the first video, but also satisfies other constraints described by the multiple frames besides body shape, such as the temporal dependencies of some information. As a result, the second video has better temporal consistency, which is beneficial for improving the body shape adjustment effect.

[0034] It should be noted that the "temporal consistency" involved in this application is used to describe the natural transition state between adjacent frames in a video, such as natural transition of light, natural transition of foreground state, natural transition of background change, and natural transition of boundary between foreground and background. This is so that the "temporal consistency" can represent the time dependence of some information in terms of time, thereby enabling the "temporal consistency" to describe the quality of the video to a certain extent. For example, high-quality videos have high temporal consistency, but videos with low temporal consistency are low-quality videos.

[0035] Furthermore, this application does not limit the executing entity of the video generation method provided in the embodiments of this application. For example, the video generation method provided in the embodiments of this application can be applied to a terminal device or a server. Alternatively, the video generation method provided in the embodiments of this application can also be implemented through a data interaction process between a terminal device and a server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0036] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0037] To better understand the technical solution provided in this application, the video generation method provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 1 As shown, the video generation method provided in this application includes S1-S4 as described below. Wherein, the... Figure 1 This is a flowchart of a video generation method provided in an embodiment of this application.

[0038] S1: Obtain the first video, in which each frame of the first video includes the target object.

[0039] The first video refers to the video that requires body shape adjustment, such as... Figure 2 The video 1 shown is used to provide information other than body shape for subsequent video generation processes, such as the pose of the foreground, the background, and the foreground identifier (Identity Document, ID).

[0040] It should be noted that this application does not limit the implementation method of the foreground ID in the preceding paragraph. For example, the foreground ID is used to describe some identifying information of the foreground presented in the first video, such as appearance, clothing, and accessories, so that the foreground ID can describe the identifying characteristics of the foreground presented in the first video, such as facial features. In addition, this application does not limit the implementation method of the foreground. For example, the foreground can be implemented as an animal, a virtual character, or an object.

[0041] In addition, in some scenarios, the first video can at least satisfy the following constraints: the first video includes multiple frames; and each frame in the first video includes a target object, so that the first video can describe the state changes of the target object over time, such as changes in posture or body shape. Here, the target object refers to the object described by the first video, such as a person, a virtual avatar, or an animal.

[0042] Furthermore, this application does not limit the target object mentioned above. For example, the target object can be used to represent the foreground described by each frame of the first video, such as a person, a digital person, an animal, or other foreground. Moreover, this application does not limit the implementation method of the target object. For example, the target object can be implemented as an animal, a virtual image, or an object.

[0043] Furthermore, for the i-th frame image in the first video, the i-th frame image refers to the image that exists in the first video and is located at the i-th position in the arrangement, and the i-th frame image can at least satisfy the following constraints: the i-th frame image includes the target object, so that the i-th frame image can at least describe the state of the target object at the acquisition time corresponding to the i-th frame image, such as posture, body shape, etc., where i is a positive integer and i ≤ the number of image frames in the first video.

[0044] Furthermore, this application does not limit the implementation of S1 above. For example, it can specifically be: receiving a video provided by a user as a first video, so that the first video can represent the video specified by the user that needs to be processed for body shape adjustment, so that body shape adjustment processing for the first video can be implemented by any implementation of the video generation method provided by this application to meet the user's video adjustment needs.

[0045] S2: For any frame of the first video, perform body shape parameter prediction processing on the image to obtain the corresponding body shape parameter prediction result. The body shape parameter prediction result is used to describe the body shape of the target object in the image.

[0046] Specifically, for the i-th frame image in the first video, the predicted body shape parameter result corresponding to the i-th frame image is obtained by performing body shape parameter prediction processing on the i-th frame image (e.g., Figure 2 The predicted parameters are obtained so that the predicted body shape can represent the body shape of the target object in the i-th frame image, where i is a positive integer and i ≤ the number of image frames in the first video.

[0047] Furthermore, this application does not limit the implementation of the above-mentioned "prediction result of body shape parameters corresponding to the i-th frame image". For example, it can be implemented using any existing or future data that can describe body shape.

[0048] In addition, to improve accuracy, the "body shape parameter prediction result corresponding to the i-th frame image" mentioned above can satisfy the following constraint: the 3D model with the body shape parameter prediction result is used to display the body shape of the target object in 3D space, so that the "body shape parameter prediction result corresponding to the i-th frame image" can more accurately describe the body shape of the target object in the i-th frame image.

[0049] The three-dimensional model is used to show the state of an object in three-dimensional space, such as its size and posture. Moreover, this application does not limit the implementation of the three-dimensional model. For example, it can be implemented using any existing or future three-dimensional model that can change its state based on parameters.

[0050] For example, in order to better improve the overall adjustment effect, the above three-dimensional model can be implemented using the SMPL (Skinned Multi-Person Linear) model, so that the body shape can be adjusted by adjusting some parameters of the three-dimensional model later.

[0051] It should be noted that SMPL, as mentioned above, is a 3D model building technique. Furthermore, SMPL can describe the state of a 3D object using two types of statistical parameters: shape parameter β and pose parameter θ. The shape parameter β includes data in 10 dimensions, allowing it to describe the overall shape of an object, such as its body shape. Each dimension of the shape parameter β can be interpreted as the object's state under a specific shape metric, such as height or build, thus enabling SMPL to describe the object's state under multiple shape metrics. Similarly, the pose parameter θ includes data in 24×3 dimensions, allowing it to describe the overall motion and posture of an object. The "24×3" refers to 23 joints plus 1 root node, and the "3" refers to the value in the axis-angle.

[0052] Based on the above four paragraphs, it can be seen that in one possible implementation, the "body shape parameter prediction result corresponding to the i-th frame image" mentioned above may include the body shape parameter in the SMPL parameter predicted for the i-th frame image, so that the "body shape parameter prediction result corresponding to the i-th frame image" can describe the state of the target object's body shape in three-dimensional space in the i-th frame image, thereby enabling the "body shape parameter prediction result corresponding to the i-th frame image" to more accurately describe the body shape of the target object presented in the i-th frame image.

[0053] It should be noted that this application does not limit the prediction method of the SMPL parameters mentioned above. For example, it can be implemented using any existing or future method that can perform SMPL parameter prediction processing on a two-dimensional image, such as a pre-built machine learning model with SMPL parameter prediction function.

[0054] Furthermore, this application does not limit the implementation of S2 described above. For example, S2 can be implemented using any existing or future method capable of predicting body shape parameters for an image, such as a method implemented using a pre-built machine learning model with body shape parameter prediction capabilities. Alternatively, to better improve the overall adjustment effect, S2 can be implemented using SMPL technology.

[0055] Therefore, in one possible implementation, S2 above can specifically be: first, perform SMPL parameter prediction processing on the i-th frame image in the first video (e.g., Figure 2 The parameter prediction process shown obtains the state parameter prediction result corresponding to the i-th frame image, so that the state parameter prediction result can be used to describe the state of the target object in the i-th frame image (such as body shape and posture). The state parameter prediction result includes body shape parameter β and posture parameter θ. Based on the state parameter prediction result, the body shape parameter prediction result corresponding to the i-th frame image is determined, so that the body shape parameter prediction result includes the body shape parameter β. The three-dimensional model with the body shape parameter prediction result belongs to the SMPL model, so that the three-dimensional model with the body shape parameter prediction result can better display the body shape of the target object in the i-th frame image in three-dimensional space. i is a positive integer, and i ≤ the number of image frames in the first video.

[0056] It should be noted that this application does not limit the implementation method of the SMPL prediction processing in the above paragraph. For example, it can be implemented using any existing or future method that can predict the SMPL parameters of an image, such as by using a pre-built machine learning model with SMPL parameter prediction function.

[0057] Based on the relevant content of S2 above, in some scenarios, such as video shooting, video beautification, or video modification, after obtaining the first video provided by the user, body shape parameter prediction processing can be performed on each frame of the first video to obtain the body shape parameter prediction results corresponding to each frame. This allows the body shape parameter prediction results corresponding to each frame to describe the body shape of the target object in the corresponding image, thus enabling the body shape parameter prediction results corresponding to these images to describe the body shape of the target object presented in the first video.

[0058] S3: For any frame of the first video, based on the body shape adjustment information specified for the target object, adjust the predicted body shape parameters of the corresponding image to obtain the body shape parameter adjustment result of the corresponding image.

[0059] The body shape adjustment information describes the body shape adjustment requirements specified by the user for the target object in the first video, such as the requirement to become 0.5 times thinner overall, so that the body shape adjustment information can represent the body shape adjustment requirements for each frame of the first video.

[0060] Furthermore, this application does not limit the implementation method of the body shape adjustment information. For example, the body shape adjustment information may include an adjustment intensity value. This adjustment intensity value can be any value within the range [-1, 1], so that the adjustment intensity value can represent the degree to which an object becomes fatter or thinner overall. For example, if the adjustment intensity value is -0.5, then the adjustment intensity value can represent that the object becomes 0.5 times thinner overall; if the adjustment intensity value is 0.5, then the adjustment intensity value can represent that the object becomes 0.5 times fatter overall.

[0061] Furthermore, this application does not limit the method of obtaining body shape adjustment information. For example, the body shape adjustment information may be obtained based on the content entered by the user through the interactive interface, such as the adjustment intensity value of -0.5.

[0062] For the i-th frame image in the first video, the body shape parameter adjustment result corresponding to the i-th frame image is obtained by adjusting the predicted body shape parameter of the i-th frame image according to the body shape adjustment information above, so that the difference between the body shape described by the body shape parameter adjustment result and the body shape described by the body shape parameter prediction result is consistent with the difference described by the body shape adjustment information, so that the difference between the body shape described by the body shape parameter adjustment result and the body shape described by the body shape parameter prediction result meets the body shape adjustment requirements described by the body shape adjustment information, where i is a positive integer and i ≤ the number of image frames in the first video.

[0063] Furthermore, this application does not limit the implementation method of S3 above. For example, it can be implemented using any existing or future parameter adjustment method, such as a method for adjusting the SMPL parameter.

[0064] For example, in some scenarios, S3 above can specifically be: first, based on the body shape adjustment information above, determine the offset of the predicted body shape parameters corresponding to each frame of the first video; then, based on the offset of the predicted body shape parameters corresponding to each frame of the video, adjust the predicted body shape parameters corresponding to each frame of the video to obtain the adjusted body shape parameters corresponding to each frame of the video.

[0065] For the i-th frame image in the first video, the offset of the predicted body shape parameter corresponding to the i-th frame image is used to describe the difference between the predicted body shape parameter corresponding to the i-th frame image and the body shape parameter expected by the body shape adjustment information above, so that the predicted body shape parameter corresponding to the i-th frame image can be adjusted according to the offset to obtain the body shape parameter expected by the body shape adjustment information.

[0066] It should be noted that, for the "body shape parameters expected by the body shape adjustment information" in the above paragraph, the "body shape parameters expected by the body shape adjustment information" refers to parameters that can meet the state adjustment requirements described by the body shape adjustment information, such as the body shape parameter adjustment result corresponding to the i-th frame image.

[0067] Furthermore, this application does not limit the implementation method of the above offset. For example, when the body shape parameter prediction result corresponding to the i-th frame image above includes data of multiple dimensions, the offset of the body shape parameter prediction result corresponding to the i-th frame image includes the offset of each dimension, so that the data of the corresponding dimension can be corrected based on the offset of each dimension to obtain the adjustment result of the data of the corresponding dimension.

[0068] As can be seen, in one possible implementation, when the predicted body shape parameters corresponding to the i-th frame image above include data in M ​​dimensions, such as 10 dimensions, the offset of the predicted body shape parameters corresponding to the i-th frame image can include the offset of the first dimension, the offset of the second dimension, ..., and the offset of the M-th dimension. The process of determining the adjusted body shape parameters corresponding to the i-th frame image can include: first, adjusting the data of the m-th dimension based on the offset of the m-th dimension to obtain the adjusted result of the m-th dimension, such as the adjusted result of the m-th dimension = the data of the m-th dimension + the offset of the m-th dimension, where m is a positive integer, m ≤ M, and M is a positive integer; then, determining the adjusted body shape parameters corresponding to the i-th frame image based on the adjusted results of the first dimension, the second dimension, ..., and the M-th dimension, so that the adjusted body shape parameters include the adjusted results of these dimensions. Here, i is a positive integer, and i ≤ the number of image frames in the first video.

[0069] Furthermore, this application does not limit the method for determining the aforementioned offset. For example, it could specifically be: mapping the aforementioned body shape adjustment information to offsets of body shape parameters according to pre-defined rules. Alternatively, it could specifically be: first searching for offsets corresponding to the aforementioned body shape adjustment information from a pre-constructed mapping relationship, and using these offsets as the offsets for the predicted body shape parameters of each frame in the first video. This mapping relationship is used to record offsets corresponding to different body shape adjustment information, such as different adjustment intensity values, so that the offsets corresponding to the user-specified body shape adjustment information can be quickly obtained subsequently using this mapping relationship.

[0070] Based on the relevant content of S3 above, in some scenarios, for the i-th frame image in the first video, after obtaining the body shape parameter prediction result corresponding to the i-th frame image, such as the 10-dimensional body shape parameter, the offset of the body shape parameter prediction result can be determined first based on the body shape adjustment information specified by the user for the target object, such as the 10-dimensional offset, so that the offset can indicate how to adjust the body shape parameter prediction result to meet the adjustment requirements described by the body shape adjustment information; then, the body shape parameter prediction result is adjusted based on the offset to obtain the body shape parameter adjustment result corresponding to the i-th frame image, so that the body shape parameter adjustment result meets the body shape adjustment requirements specified by the user for the target object, where i is a positive integer and i ≤ the number of image frames in the first video.

[0071] S4: Based on the first video and the body shape parameter adjustment results corresponding to each frame of the first video, generate a second video. The i-th frame of the second video is used to represent the body shape adjustment result of the i-th frame of the first video. The i-th frame of the second video is generated based on multiple frames of images arranged sequentially in the first video and the body shape parameter adjustment results corresponding to the multiple frames. The distance between the arrangement position of each frame of the multiple frames in the first video and the arrangement position of the i-th frame of the first video in the first video is not greater than a preset distance threshold. i is a positive integer, i≤ the number of image frames in the second video or the number of image frames in the first video.

[0072] The second video represents the result of body shape adjustments made to the first video based on the body shape adjustment information provided above, such as... Figure 2 The second video 2 shown is configured such that the second video and the first video can at least satisfy the following constraints: the second video and the first video are in a time-aligned state, and the body type described in the second video is different from the body type described in the first video.

[0073] It can be seen that, in one possible implementation, the second video mentioned above can at least satisfy the following constraints: the i-th frame image in the second video is used to represent the body shape adjustment result for the i-th frame image in the first video, so that the difference between the body shape described by the i-th frame image in the second video and the body shape described by the i-th frame image in the first video satisfies the difference described by the body shape adjustment information mentioned above as much as possible, where i is a positive integer and i ≤ the number of image frames in the second video.

[0074] In addition, in one possible implementation, to better improve the effect, the second video mentioned above can at least satisfy the following constraints: the i-th frame image in the second video is used to represent the body shape adjustment result based on the i-th frame image in the first video, the body shape described by the i-th frame image in the second video is different from the body shape described by the i-th frame image in the first video, and the other information described by the i-th frame image in the second video other than the body shape is consistent with the other information described by the i-th frame image in the first video other than the body shape, where i is a positive integer and i ≤ the number of image frames in the second video. This can effectively reduce the interference caused by body shape modification to other image information, thereby improving the effect.

[0075] Furthermore, in one possible implementation, in order to better improve the effect, the second video mentioned above can at least satisfy the following constraints: the prediction results of the body shape parameters corresponding to each frame of the second video are consistent, so as to ensure that the objects described by different images in the second video are as consistent as possible in terms of body shape, thereby helping to avoid adverse effects caused by large differences in the body shape of the objects described by different images in the generated video, such as poor visual effects, and thus helping to improve the body shape modification effect.

[0076] It should be noted that this application does not limit the implementation of the concept of "maintaining consistency". For example, in some scenarios, the concept of "maintaining consistency" is used to mean that they are completely identical. As an example, if information 1 and information 2 maintain consistency, it can mean that information 1 and information 2 are completely identical.

[0077] For example, in some scenarios, the concept of "maintaining consistency" mentioned above is used to indicate a relatively high degree of similarity. As an example, if information 1 and information 2 maintain consistency, it can be said that the degree of similarity between information 1 and information 2 is relatively high, such as if the degree of similarity between information 1 and information 2 is higher than a preset similarity threshold.

[0078] Furthermore, to better ensure temporal consistency, the second video mentioned above should at least satisfy the following constraints: the i-th frame in the second video represents the body shape adjustment result based on the i-th frame in the first video; the i-th frame in the second video is generated based on multiple frames arranged sequentially in the first video and the corresponding body shape parameter adjustment results; and the distance between the arrangement position of each frame in the multiple frames in the first video and the arrangement position of the i-th frame in the first video is no greater than a preset distance threshold, where i is a positive integer and i ≤ the number of frames in the second video. This preset distance threshold can be set according to the actual application scenario.

[0079] As can be seen, when the preset distance threshold is Y, where Y is a positive integer, if the i-th frame image in the second video is used to represent the body shape adjustment result based on the i-th frame image in the first video, then the i-th frame image in the second video can be the body shape adjustment result based on the iY-th frame image in the first video and the corresponding body shape parameter, the i-Y+1-th frame image in the first video and the corresponding body shape parameter adjustment result, ... (and so on), and the i+Y-th frame image in the first video and the corresponding body shape parameter. The parameter adjustment results in the generation of the i-th frame image in the second video, which not only references the relevant information of the i-th frame image in the first video, but also the relevant information of some neighboring images of the i-th frame image in the first video. This results in the i-th frame image in the second video exhibiting better temporal characteristics, such as trends and time dependencies, with i being a positive integer and i ≤ the number of frames in the second video. This ensures that the final generated second video has good temporal consistency.

[0080] In addition, in some scenarios, to better improve temporal consistency, the second video mentioned above can at least satisfy the following constraints: the i-th frame image in the second video is used to represent the body shape adjustment result based on the i-th frame image in the first video. The i-th frame image in the second video is generated based on all images in the first video and the body shape parameter adjustment results corresponding to all images. This ensures that the generation process of the i-th frame image in the second video not only refers to the relevant information of the i-th frame image in the first video, but also refers to the relevant information of other images in the first video. This results in the i-th frame image in the second video exhibiting better temporal characteristics with other images in the second video, such as change trends and time dependencies, which is beneficial to improving the body shape adjustment effect.

[0081] Furthermore, this application does not limit the implementation of S4 described above. For example, S4 can be implemented using a target model. This target model is used to generate new videos based on its input data; and this application does not limit the target model. For example, the target model can be implemented using any existing or future diffusion model. Therefore, in one possible implementation, the target model may include at least one UNet network, such as... Figure 2 or Figure 3 The two UNet networks shown.

[0082] Furthermore, to further improve the body shape adjustment effect, this application also provides a possible implementation of the target model described above. For example, the target model can at least satisfy the following constraints: the target model includes at least one processing unit (such as...). Figure 4 The processing unit shown includes an image-level processing module and a timing module. The image-level processing module is used to implement separate processing procedures for each frame of the above multi-frame images, and the timing module is used to perform attention processing on the execution results of the separate processing procedures for the multi-frame images in the time direction.

[0083] Here, a processing unit refers to a unit within the target model that is used to process multiple images, such as... Figure 4 The processing unit shown may include Figure 3 The units of modules B1 and T are shown.

[0084] It should be noted that, for Figure 3 In network 2, module B1 is used to implement downsampling processing for the first size, and the output data of module B1 is of the first size; module B2 is used to implement downsampling processing for the second size, and the output data of module B2 is of the second size; ... (and so on); B N The module is used to implement downsampling processing for the Nth size, and the B N The size of the module's output data is this Nth size; B N+1 The module is used to implement upsampling processing for the Nth size, and the B N+1 The size of the module's input data is this Nth size; B N+2 The module is used to implement upsampling processing for the (N-1)th size, and this B N+2 The input data size of the module is the (N-1)th size; ... (and so on); B 2N The module is used to implement upsampling processing for the first size, and the B 2N The input data size of the module is the first type; T module represents a temporal module. Additionally, for... Figure 3In network 1, module A1 is used to implement downsampling processing for the first size, and the output data of module A1 is of the first size; module A2 is used to implement downsampling processing for the second size, and the output data of module A2 is of the second size; ... (and so on); A N The module is used to implement downsampling processing for the Nth size, and this A N The size of the module's output data is this Nth size; A N+1 The module is used to implement upsampling processing for the Nth size, and this A N+1 The size of the module's input data is this Nth size; A N+2 The module is used to implement upsampling processing for the (N-1)th size, and this A N+2 The input data of the module is of the (N-1)th size; ... (and so on); A 2N The module is used to implement upsampling processing for the first size, and this A 2N The input data size of the module is the first size; T module represents a temporal module.

[0085] In addition, the above processing unit can at least satisfy the following constraints: the processing unit includes an image-level processing module (such as... Figure 4 The processing module shown Figure 3 B1-B shown 2N any one or Figure 3 A1-A shown 2N (any one) and timing modules (such as Figure 4 The timing module shown or Figure 3 The T module shown enables the processing unit to perform not only individual processing of each frame in the first video, but also joint processing of multiple frames in the first video, thereby achieving better timing consistency.

[0086] For the image-level processing module mentioned above, such as Figure 4 In the context of the processing module shown, the image-level processing module refers to the module present in the aforementioned processing unit that performs image-level related processing, enabling the image-level processing module to process data on an image-by-image basis. This allows the image-level processing module to process input data (such as...) Figure 4 The F-frame data shown in the figure realizes the image-level processing process, so that when the image-level processing module is used to realize the correlation processing of multiple frames of images in the first video, the image-level processing module can realize the correlation processing of each frame of images in parallel or serial manner, so as to ensure that the correlation processing of different images is independent and does not interfere with each other.

[0087] Furthermore, this application does not limit the implementation of the above image-level processing module. For example, when the above target model is implemented using a diffusion model, if the target model includes at least one UNet network, the image-level processing module can be implemented using each upsampling module and each downsampling module in the UNet network.

[0088] For the timing module mentioned above, such as Figure 4 In the context of the timing module shown, this timing module refers to the module within the aforementioned processing unit that performs video-level (or sequence-level) related processing. This allows the timing module to process data on a per-image-sequence basis, enabling it to be used for input data or output data from the image-level processing module within the processing unit (e.g., ...). Figure 4 The data shown 1) implements video-level processing, so that when the timing module is used to perform related processing on multiple frames of images (e.g., F frames) in the first video, the timing module can process the multiple frames as a whole, so as to ensure that the related processing of each frame of the multiple frames not only refers to the image itself, but also to other images, which is conducive to better achieving timing consistency.

[0089] Furthermore, this application does not limit the implementation method of the timing module mentioned above. For example, it can be implemented using any neural network capable of achieving timing consistency optimization processing.

[0090] Furthermore, to improve performance, the aforementioned timing module may include an integration network, a first transformation network, at least one self-attention network, a second transformation network, and a splitting network, thereby enhancing the timing consistency optimization performance of the module. For ease of understanding, each network is described below.

[0091] For the integration network in the timing module mentioned above, this integration network is used to integrate the input data of the timing module (or, including the output data of the image-level processing module in the processing unit of the timing module) to obtain an integration result. The size of the integration result is different from the size of the input data of the timing module, and the integration result can represent the input data of the timing module as a whole for subsequent processing. Here, the input data of the timing module refers to the data input to the timing module including the integration network, such as... Figure 4 The data shown is 1. To facilitate understanding, the following explanation is provided with an example.

[0092] As an example, for a timing module, when the size of the input data of the timing module is [b, c, F, h, w], where b represents the number of batches, c represents the number of channels, F represents the number of image frames, h represents the image height, and w represents the image width, the integration network in the timing module is used to integrate the input data of the timing module to obtain an integration result, so that the size of the integration result is [b×h×w, F, c], so that the integration result can represent the input data of the timing module as a whole for subsequent processing.

[0093] Furthermore, this application does not limit the above-described integrated network; for example, it can employ reshape, such as... Figure 4 The Reshape shown is implemented.

[0094] For the first transformation network in the timing module above, such as Figure 4 In the Project in module shown, the first transformation network is used to perform representation transformation processing on the output data of the integration network in the time-series module. Specifically, when the output data of the integration network is the integration result mentioned above, the first transformation network converts the integration result from a first representation to a second representation, obtaining a first transformation result. This first transformation result describes the information described by the integration result according to the representation method of video features. Therefore, the first transformation result not only represents the information described by the integration result but also represents the unique characteristics of the video features. Consequently, the self-attention processing implemented based on the first transformation result can better capture the unique characteristics of the video features, thus improving the video generation effect. Specifically, the first representation method is used to describe image features, making it the representation method possessed by the image features, thereby enabling the first representation method to represent the unique characteristics of the image features. The second representation method is used to describe video features, making it the representation method possessed by the video features, thereby enabling the second representation method to represent the unique characteristics of the video features.

[0095] Furthermore, this application does not limit the implementation of the first conversion network described above. For example, the first conversion network can be implemented using a fully connected architecture.

[0096] For at least one self-attention network in the temporal module above, such as Figure 4As shown in the Self-attention×4 example, these self-attention networks are used to perform self-attention processing on the output data of the first transformation network mentioned above, so that these self-attention networks can achieve the "attention processing in the time direction" mentioned above. This allows these self-attention networks to capture the temporal dependencies between features at the same position on the time axis from the output data, thereby making the data obtained based on the temporal dependencies (such as the self-attention processing results below) have better temporal consistency.

[0097] As can be seen, when the output data of the first transformation network in the above-mentioned timing module is the first transformation result, at least one self-attention network in the timing module can be used to: perform self-attention processing on the first transformation result to obtain the self-attention processing result, so that the self-attention processing result presents better timing consistency.

[0098] Furthermore, this application does not limit the working principle of the at least one self-attention network mentioned above. For example, when the at least one self-attention network includes D self-attention networks, where D is a positive integer, the working principle of the at least one self-attention network can be as follows: the first self-attention network performs self-attention processing on the output data of the first transformation network mentioned above to obtain the output data of the first self-attention network; the second self-attention network performs self-attention processing on the output data of the first self-attention network to obtain the output data of the second self-attention network; the third self-attention network performs self-attention processing on the output data of the second self-attention network to obtain the output data of the third self-attention network; ... (and so on); the Dth self-attention network performs self-attention processing on the output data of the (D-1)th self-attention network to obtain the output data of the Dth self-attention network, and the output data of the Dth self-attention network is regarded as the output data of the at least one self-attention network (as described above) for subsequent processing.

[0099] Furthermore, this application does not limit the implementation method of the self-attention network described above. For example, the self-attention network can be implemented using any existing or future self-attention network.

[0100] For the second conversion network in the timing module above, such as Figure 4In the Project out module shown, the second transformation network is used to perform an inverse transformation of the representation mode on the output data of at least one self-attention network in the time sequence module. Specifically, the inverse transformation of the representation mode can be: when the output data of the at least one self-attention network is the self-attention processing result mentioned above, the second transformation network is used to convert the self-attention processing result from the second representation mode to the first representation mode to obtain the second transformation result, so that the second transformation result can describe the information described by the self-attention processing result according to the representation mode of the image features.

[0101] Furthermore, this application does not limit the implementation of the second conversion network described above. For example, the second conversion network can be implemented using a fully connected architecture.

[0102] For the splitting network in the timing module mentioned above, this splitting network is used to split the output data of the second transformation network in the timing module. Specifically, this splitting process can be as follows: when the output data of the second transformation network is the second transformation result mentioned above, the splitting network is used to split the second transformation result to obtain a splitting result, such as... Figure 4 The data shown in Figure 2 is used to make the size of the split result the same as the size of the input data of the timing module, so that the size of the split result can be [b, c, F, h, w].

[0103] Furthermore, this application does not limit the network to the aforementioned splitting method; for example, it can employ reshape, such as... Figure 4 The Reshape shown is implemented.

[0104] Based on the above content regarding the timing module, in one possible implementation, when the input data of the timing module has a size of [b, c, F, h, w], the working principle of the timing module can be as follows: First, reshape the input data of the timing module to the size of [b×h×w, F, c] to obtain the integrated result; then, convert the integrated result from image feature representation to video feature representation through a fully connected method to obtain the first transformation result, so that the size of the first transformation result remains unchanged; then, perform the first transformation through multiple self-attention networks (such as 4 self-attention networks). The results are processed to obtain self-attention processing results, which are designed to at least represent the temporal dependencies between features captured by these self-attention networks at the same location on the time axis. Next, the self-attention processing results are converted from video feature representation to image feature representation using a fully connected approach, resulting in a second transformation result. Finally, the second transformation result is reshaped to the size [b, c, F, h, w] to obtain the output data of the temporal module (as shown in the splitting results above). This enables temporal consistency optimization processing for multiple frames of images.

[0105] In addition, in some scenarios, in order to better improve the timing consistency optimization effect, this application also provides a possible implementation of the timing module mentioned above. In this way, when the processing unit includes an image-level processing module and the timing module, and the image-level processing module is used to implement the individual processing process for each frame of the multi-frame images in the first video mentioned above, the timing module can at least satisfy the following constraints: the timing module is used to perform attention processing on the execution result of the individual processing process for the multi-frame images in the time direction, and the attention processing is also implemented based on the time information of each frame of the multi-frame images; wherein, for any frame of the multi-frame images, the time information of the image is used to describe the arrangement position of the image in the first video.

[0106] For the i-th frame image in the first video, the time information of the i-th frame image is used to describe the arrangement position of the i-th frame image in the first video (or the acquisition time corresponding to the i-th frame image in the first video); and this application does not limit the implementation of the time information. For example, it can be implemented using the arrangement position to ensure that the time information involved in the inference process and the time information involved in the training process are consistent, thereby helping to improve the body shape adjustment effect.

[0107] For example, to improve the effect, the time information of the i-th frame image mentioned above can be obtained by performing position encoding processing (such as sine position encoding processing) on ​​the arrangement position of the i-th frame image in the first video, so that the time information can better represent the spatiotemporal position of the i-th frame image in the first video.

[0108] As can be seen, in one possible implementation, the time information of the i-th frame image in the first video may include the position encoding result of the i-th frame image's position in the first video, so that the time information can better describe the spatiotemporal position of the i-th frame image in the first video.

[0109] Furthermore, this application does not limit the way the timing module uses the time information in the preceding text. For example, when the preceding text "multiple frames of images arranged sequentially in the first video" includes all images in the first video, and the preceding processing unit includes an image-level processing module and the timing module, and the image-level processing module is used to implement separate processing for each frame of the multi-frame images, the execution result of the separate processing of the i-th frame of the first video can be fused with the time information of the i-th frame, such as by splicing, to obtain the fused result of the i-th frame, where i is a positive integer and i ≤ the number of image frames in the first video; then the fused result of the multi-frame images is input into the timing module so that the timing module performs timing consistency optimization processing on these fused results.

[0110] Based on the above description of the processing unit, in one possible implementation, the target model described above can at least satisfy the following constraints: the target model includes at least one image-level processing module and a timing module corresponding to each image-level processing module, and for any image-level processing module, the input data of the timing module corresponding to that image-level processing module includes the output data of that image-level processing module. For ease of understanding, two examples are provided below.

[0111] Example 1: In some scenarios, the target model mentioned above may include an image-level processing module and a timing module; and the working principle of the target model can be as follows: first, the image-level processing module processes the input data of the target model to obtain the output data of the image-level processing module; then, the timing module processes the output data of the image-level processing module to obtain the output data of the timing module; then, based on the output data of the timing module, the output data of the target model is determined, as in the second video above.

[0112] It should be noted that this application does not limit the implementation method of the image-level processing module in the preceding paragraph. For example, when the target model described above is implemented using at least one UNet network, the image-level processing module may include at least one UNet network. The UNet network can be any existing or future UNet network capable of image-level processing.

[0113] Example 2: In some scenarios, the target model mentioned above may include Q image-level processing modules and Q time-series modules. The working principle of this target model can be as follows: First, the first image-level processing module processes the input data of the target model to obtain its output data; then, the first time-series module processes the output data of the first image-level processing module to obtain its output data; then, the second image-level processing module processes the output data of the first time-series module to obtain its output data; and finally, the second time-series module processes the output data of the second image-level processing module... The process involves several steps: first, processing the data to obtain the output data of the second timing module; then, processing the output data of the third image-level processing module to obtain its own output data; and so on. Next, the Qth image-level processing module processes the output data of the (Q-1)th timing module to obtain its own output data; and so on. Here, Q is a positive integer.

[0114] It should be noted that this application does not limit the implementation of the q-th image-level processing module in the above paragraph. For example, when the target model mentioned above is implemented with the help of the UNet network, the q-th image-level processing module can refer to the module that exists in the UNet network and is located at the q-th position, such as an upsampling module or a downsampling module, where q is a positive integer, q≤Q, and Q is a positive integer.

[0115] As can be seen, in one possible implementation, the target model described above can at least satisfy the following constraints: the target model includes at least one processing unit, and the at least one processing unit includes multiple upsampling modules, a timing module corresponding to each upsampling module, multiple downsampling modules, and a timing module corresponding to each downsampling module; wherein, for any upsampling module, the timing module corresponding to the upsampling module is used to perform attention processing on the output data of the upsampling module in the temporal direction; for any downsampling module, the timing module corresponding to the downsampling module is used to perform attention processing on the output data of the downsampling module in the temporal direction, thus enabling temporal consistency optimization processing at as many resolutions as possible, thereby facilitating the expansion of the receptive field and thus better achieving temporal consistency.

[0116] Furthermore, this application does not limit the implementation of the target model shown in the above paragraph. For ease of understanding, two examples are provided below.

[0117] Example 1: In some scenarios, if the target model described above is implemented using a single UNet network, then the target model may include multiple modules arranged sequentially, and these multiple modules include downsampling modules of the first size (such as...). Figure 4 The B1 module shown), the timing module corresponding to the first type of downsampling module, and the second type of downsampling module (such as...) Figure 4 The B2 module shown), the timing module corresponding to the second size downsampling module, ..., the Nth size downsampling module (such as... Figure 4 B shown N (module), the timing module corresponding to the downsampling module of the Nth size, and the upsampling module of the Nth size (such as...) Figure 4 B shown N+1 The upsampling module for the Nth size, the timing module corresponding to the upsampling module for the (N-1)th size, the timing module corresponding to the upsampling module for the (N-1)th size, ..., the upsampling module for the second size (e.g. Figure 4 B shown 2N-1 The second type of upsampling module), the timing module corresponding to the second type of upsampling module, and the first type of upsampling module (such as...) Figure 4 B shown 2N (Module).

[0118] For the downsampling module of the nth size mentioned above, the downsampling module of the nth size can be used to implement downsampling processing of the nth size; the size of the output data of the downsampling module of the nth size is the nth size; and the timing module corresponding to the downsampling module of the nth size is used to perform timing consistency optimization processing on the output data of the downsampling module of the nth size at the resolution described by the nth size, where n is a positive integer and n≤N.

[0119] Furthermore, this application does not limit the implementation method of the input data of the downsampling module of the nth size mentioned above. For example, when n=1, the input data of the downsampling module of the nth size may include the input data of the target model; when n≥2, the input data of the downsampling module of the nth size may include the output data of the timing module corresponding to the downsampling module of the (n-1)th size.

[0120] Furthermore, in some scenarios, such as those requiring high model training performance, when n≥2, the input data of the downsampling module of the nth size mentioned above can include the sum or residual between the output data of the timing module corresponding to the downsampling module of the (n-1)th size and the output data of the downsampling module of the (n-1)th size, such as... Figure 4 Data 3 is shown.

[0121] The upsampling module for the nth size mentioned above is used to implement upsampling processing for the nth size; the input data size of the upsampling module for the nth size is the nth size; and the timing module corresponding to the upsampling module for the nth size is used to perform timing consistency optimization processing on the resolution described by the (n-1)th size for the output data of the upsampling module for the nth size, where n is a positive integer and n≤N.

[0122] Furthermore, this application does not limit the implementation method of the input data of the upsampling module of the nth size mentioned above. For example, when n = N, the input data of the upsampling module of the nth size may include the output data of the timing module corresponding to the downsampling module of the Nth size mentioned above; when n ≤ N-1, the input data of the upsampling module of the nth size may include the output data of the timing module corresponding to the upsampling module of the (n+1)th size.

[0123] Furthermore, in some scenarios, such as those requiring high model training performance, when n = N, the input data of the upsampling module of the nth size mentioned above can include the sum or residual between the output data of the timing module corresponding to the downsampling module of the Nth size and the output data of the downsampling module of the Nth size; when n ≤ N-1, the input data of the upsampling module of the nth size can include the sum or residual between the output data of the timing module corresponding to the upsampling module of the (n+1)th size and the output data of the upsampling module of the (n+1)th size.

[0124] Example 2: In some scenarios, if the target model described above is implemented using two UNet networks, then the target model can at least satisfy the following constraint: the target model includes a first network (such as...). Figure 3 Network 1) and the second network (as shown) Figure 3 Network 2 shown; the network structure of the first network is the same as that of the second network; the first network includes multiple first modules arranged in sequence, and the second network includes multiple second modules arranged in sequence; the output data of some or all modules in the first network is injected into the corresponding modules in the second network through cross-attention; the input data of the first network is determined based on the first video; the input data of the second network is determined based on the body shape parameter adjustment results corresponding to each frame of the first video.

[0125] It should be noted that the implementation method of "multiple first modules in a sequential arrangement" in the previous paragraph is similar to the implementation method of "multiple modules in a sequential arrangement" in the previous text, and the implementation method of "multiple second modules in a sequential arrangement" in the previous paragraph is also similar to the implementation method of "multiple modules in a sequential arrangement" in the previous text. For the sake of brevity, it will not be repeated here.

[0126] Furthermore, this application does not limit the working principle of the first network described above. For example, when the input data of the first network is determined based on the first video, the working principle of the first network (such as...) Figure 2 The working principle of the network 1 shown may include: after obtaining the image features of the i-th frame image in the first video, the first network processes the image features to obtain the output data of each module in the first network; and the semantic features of the i-th frame image in the first video are introduced into some or all modules (such as at least one upsampling module) in the first network through cross-attention, so that some or all modules can perform corresponding processing based on the semantic features.

[0127] For the i-th frame of the first video above, the image features of the i-th frame are obtained by performing image feature extraction processing on the i-th frame, such as... Figure 2The feature extraction method shown in step 1 is used to obtain image features of the i-th frame image to describe the information carried by the i-th frame image, such as local information of each pixel and some global information of the i-th frame image; the semantic features of the i-th frame image are obtained by performing semantic feature extraction processing on the i-th frame image, such as... Figure 2 The Contrastive Language-Image Pre-Training (CLIP) model shown is used to obtain the semantic features of the i-th frame image to describe the semantic information carried by the i-th frame image, such as information like image architecture, where i is a positive integer and i ≤ the number of image frames in the first video.

[0128] It should be noted that this application does not limit the implementation method of "image feature extraction processing" in the above paragraph. For example, it can be implemented using any existing or future encoder with image feature extraction function. Similarly, this application does not limit the implementation method of "semantic feature extraction processing" in the above paragraph. For example, it can be implemented using any existing or future encoder with semantic feature extraction function, such as the image encoder in the CLIP model.

[0129] Furthermore, this application does not limit the working principle of the second network described above. For example, when the second network is used to implement noise reduction processing, and the input data of the second network is determined based on the body shape parameter adjustment results corresponding to each frame of the first video, the working principle of the second network (e.g.) Figure 2The working principle of network 2 shown may include: firstly, based on the body shape parameter adjustment result and the pose parameter prediction result (such as pose parameter θ) corresponding to the i-th frame image in the first video, determining the state parameter adjustment result corresponding to the i-th frame image, so that the state parameter adjustment result includes the body shape parameter adjustment result and the pose parameter prediction result; then, visualizing the state parameter adjustment result corresponding to the i-th frame image to obtain the visualization result corresponding to the i-th frame image, so that the visualization result can present the body shape and pose described by the state parameter adjustment result in an image manner; then, based on the body shape parameter adjustment result and the pose parameter prediction result (such as pose parameter θ) corresponding to the i-th frame image in the first video, determining the state parameter adjustment .... The visualization processing result corresponding to the i-th frame image is subjected to state feature extraction processing to obtain the state feature extraction result corresponding to the i-th frame image, so that the state feature extraction result can better describe the body shape and posture described by the state parameter adjustment result; then, the state feature extraction result corresponding to the i-th frame image and the randomly generated noise image corresponding to the i-th frame image are stitched together to obtain the stitched result corresponding to the i-th frame image, where i is a positive integer, i ≤ the number of image frames in the first video; then, the stitched results corresponding to all images in the first video are input into the second network so that the second network can process these stitched results, such as Figure 3 The processing involved in Network 2, etc. Among them, for some or all modules in the second network (e.g., at least one upsampling module), the module can be used at least to perform cross-attention processing based on the semantic features of each frame of the first video and the output data of the corresponding module in the first network above, so as to improve the processing performance of the module.

[0130] It should be noted that the pose parameter prediction result corresponding to the i-th frame image in the first video above is obtained by performing pose parameter prediction processing (such as SMPL) on the i-th frame image, so that the pose parameter prediction result can be used to describe the pose of the target object in the i-th frame image. Furthermore, this application does not limit the implementation method of the pose parameter prediction processing. In addition, this application does not limit the implementation method of the "visualization processing" mentioned above. For example, when the state parameter adjustment result corresponding to the i-th frame image is implemented using SMPL parameters, the visualization processing can specifically be as follows: first, a three-dimensional model, such as an SMPL model, is constructed based on the state parameter adjustment result, so that the three-dimensional model can present the body shape and pose described by the state parameter adjustment result in three-dimensional space; then, two-dimensional image capture processing is performed on the three-dimensional model to obtain the visualization processing result corresponding to the i-th frame image, so that the visualization result can present the body shape and pose described by the state parameter adjustment result in an image manner, thereby enabling the visualization result to not only represent the body shape and pose described by the state parameter adjustment result, but also describe some spatial information corresponding to the body shape and pose, which is beneficial to improving the video generation effect.

[0131] Based on the above content about the target model, it can be seen that in some scenarios, when S4 above is implemented using the target model, S4 may specifically include steps 11-14 below.

[0132] Step 11: Perform image feature extraction processing on the i-th frame of the first video, such as... Figure 2 Feature extraction 1, as shown, obtains the image features of the i-th frame image, so that these image features can be used to describe the image information carried by the i-th frame image, where i is a positive integer, and i ≤ the number of image frames in the first video (e.g., ...). Figure 3 The F shown.

[0133] Step 12: Perform semantic feature extraction processing on the i-th frame image in the first video, such as... Figure 2 The CLIP shown obtains the semantic features of the i-th frame image, so that the semantic features can be used to describe the semantic information carried by the i-th frame image, where i is a positive integer and i ≤ the number of image frames in the first video (e.g., ...). Figure 3 The F shown.

[0134] Step 13: Based on the body shape parameter adjustment results corresponding to the i-th frame image in the first video, determine the state feature extraction results corresponding to the i-th frame image, such as... Figure 2 The output data of feature extraction 2 shown, along with the state feature extraction result corresponding to the i-th frame image and the randomly generated noise image for the i-th frame image, are stitched together to obtain the stitching result corresponding to the i-th frame image, where i is a positive integer and i ≤ the number of image frames in the first video (e.g., ...). Figure 3 The F shown.

[0135] It should be noted that this application does not limit the execution relationship between steps 11 to 13 above. For example, the three can be executed simultaneously, or the three can be executed in a certain order.

[0136] Step 14: From the first network in the target model (e.g., Figure 2 or Figure 3 The network 1 shown processes the image features and semantic features of all images in the first video to obtain the output data of each module in the first network, and the output data of the second network in the target model (such as...) Figure 2 or Figure 3 Network 2) as shown processes the stitching results of all images in the first video, the semantic features of all images, and the output data of some or all modules in the first network to obtain the second video, such as... Figure 2 Video 2 is shown.

[0137] Based on the content of steps 11 to 14 above, it is known that in some scenarios, for the first video, such as Figure 2 Taking the video 1 shown as an example, firstly, the image features of each frame in the first video, the semantic features of each frame in the first video, and the splicing result of each frame in the first video are obtained; then, these image features, these semantic features, and these splicing results are input into the target model, so that the target model can generate and output the second video based on these image features, these semantic features, and these splicing results, as shown. Figure 2 Video 2 is shown.

[0138] The timing module in the target model has timing consistency optimization performance, which enables the second video generated using the target model to present a better timing consistency effect. This ensures that the second video does not contain various factors that affect timing consistency, such as background jumps, sudden changes in body shape, and sudden changes in posture, thereby improving the video generation effect and thus improving the body shape adjustment effect.

[0139] In addition, this application achieves temporal consistency optimization for each resolution in the UNet network by adding a timing module after each module in the UNet network. This helps to expand the receptive field, thereby improving the video generation effect and the body shape adjustment effect.

[0140] Furthermore, this application introduces temporal information of each frame of the first video, such as sinusoidal position coding, into the self-attention processing network of each temporal module, so that the self-attention processing network can focus on the spatiotemporal position of these images, thereby improving the temporal consistency optimization effect and thus improving the body shape adjustment effect.

[0141] Based on the relevant content of S1 to S4 above, the video generation method provided in this application firstly acquires a first video so that each frame of the first video includes the target object; secondly, for any frame of the first video, body shape parameter prediction processing is performed on the image to obtain the corresponding body shape parameter prediction result, so that the body shape parameter prediction result can be used to describe the body shape of the target object in the image; then, for any frame of the first video, according to the body shape adjustment information specified for the target object (such as losing 20% ​​or other information), the corresponding body shape parameter prediction result is adjusted to obtain the corresponding body shape parameter adjustment result; finally, based on the first video and the corresponding body shape parameter adjustment results of each frame of the first video, a second video is generated so that the second video can be used to represent the body shape adjustment result for the first video, such that the i-th frame of the second video is used to represent the body shape adjustment result for the i-th frame of the first video, where i is a positive integer and i ≤ the number of image frames in the second video or the number of image frames in the first video.

[0142] In this process, the i-th frame image in the second video is generated based on multiple frames arranged sequentially in the first video and the body shape parameter adjustment results corresponding to those multiple frames. Furthermore, the distance between the arrangement position of each frame image in the first video and the arrangement position of the i-th frame image in the first video is not greater than a preset distance threshold. This ensures that the i-th frame image in the second video not only satisfies the body shape constraints described by the body shape parameter adjustment results corresponding to the i-th frame image in the first video, but also satisfies other constraints described by the multiple frames besides body shape, such as the temporal dependencies of some information. As a result, the second video has better temporal consistency, which is beneficial for improving the body shape adjustment effect.

[0143] In addition, in some scenarios, the timing module can be configured to perform timing consistency optimization processing on a limited number of images. Therefore, in order to better meet this requirement, this application also provides a possible implementation of the video generation method described above. In this implementation, the video generation method may include steps 21-29 below.

[0144] Step 21: Obtain a long video provided by the user, where the number of image frames is greater than a preset frame count threshold. This preset frame count threshold describes the maximum number of image frames that the timing module described above can process simultaneously.

[0145] Step 22: Determine multiple video segments arranged sequentially from the long video, such that the number of image frames in each video segment is equal to a preset frame number threshold, and that the multiple video segments can completely cover all images in the long video. Also, ensure that the first frame of the video segment that is arranged earlier in the long video is captured earlier than the first frame of the video segment that is arranged later in the long video.

[0146] It should be noted that this application does not limit the implementation of step 22 above. For example, it can be implemented by means of a sliding window.

[0147] As can be seen, in one possible implementation, when the window size of the sliding window is a preset frame number threshold (e.g., 24) and the sliding stride of the sliding window is a preset stride value (e.g., 20), step 22 can specifically be: using the sliding window, determining multiple video segments arranged sequentially from the long video, such as the first video segment including the first frame image to the 24th frame image, the second video segment including the 21st frame image to the 44th frame image, the third video segment including the 41st frame image to the 64th frame image, ... (and so on), so that the number of image frames in each video segment is equal to the preset frame number threshold, and the number of image frames between the first frames of any two adjacent video segments is the preset stride value, and the first frame image of the video segment with a smaller arrangement number is positioned earlier in the long video than the first frame image of the video segment with a larger arrangement number.

[0148] Additionally, in some scenarios, for the long video mentioned above, after obtaining some video segments from the long video, if the number of frames of the remaining images in the long video is less than a preset frame number threshold, in order to better ensure the timing consistency optimization effect, the missing images can be supplemented with completely black images so that the video segment with the last position in the final arrangement includes these remaining images as well as some completely black images, and makes the number of frames of the images in the video segment with the last position in the final arrangement equal to the preset frame number threshold.

[0149] Furthermore, in some scenarios, for the long video mentioned above, after obtaining some video segments from the long video, if the number of frames of the remaining images in the long video is less than a preset frame number threshold, in order to better ensure the temporal consistency optimization effect, these remaining images, as well as some images in the long video that are relatively close to these remaining images, can be used to construct the video segment with the latest arrangement position. This ensures that the video segment with the latest arrangement position includes the multiple frames of images with the latest arrangement position in the long video, and the number of frames of the multiple frames of images with the latest arrangement position is equal to the preset frame number threshold. This makes the number of frames of the overlapping images between the video segment with the latest arrangement position and the video segment with the second-to-last arrangement position greater than the difference between the preset frame number threshold and the preset step size value mentioned above (such as 4).

[0150] Step 23: Initialize the first video using the video segment that appears first in the order, so that the initial value of the first video segment includes the video segment that appears first in the order.

[0151] Step 24: For any frame of the first video, perform body shape parameter prediction processing on the image to obtain the corresponding body shape parameter prediction result. The body shape parameter prediction result is used to describe the body shape of the target object in the image.

[0152] It should be noted that for the relevant content of step 24, please refer to the relevant content of S2 above.

[0153] Step 25: For any frame of the first video, adjust the predicted body shape parameters of the image according to the body shape adjustment information specified for the target object, and obtain the body shape parameter adjustment result of the image.

[0154] It should be noted that for details regarding step 25, please refer to the relevant content in S3 above.

[0155] Step 26: Based on the target model, the first video, and the body shape parameter adjustment results corresponding to each frame of the first video, generate a second video corresponding to the first video. The i-th frame of the second video is used to represent the body shape adjustment result for the i-th frame of the first video. The i-th frame of the second video is generated based on multiple frames arranged sequentially in the first video and the body shape parameter adjustment results corresponding to the multiple frames. The distance between the arrangement position of each frame in the multiple frames in the first video and the arrangement position of the i-th frame in the first video is not greater than a preset distance threshold. i is a positive integer, and i ≤ the number of image frames in the second video or the number of image frames in the first video.

[0156] It should be noted that for details regarding step 26, please refer to the relevant content in S4 above.

[0157] Step 27: Determine if there are any untraversed video segments among the multiple video segments above. If so, it can be determined that some video segments have not yet undergone size adjustment, so step 28 below can be executed; if not, it can be determined that all video segments have completed size adjustment, so step 29 below can be executed.

[0158] Step 28: If there are untraversed video segments among the multiple video segments above, then use the video segment with the highest position in the untraversed video segments to update the first video, so that the updated first video includes the video segment with the highest position in the untraversed video segments, and return to continue executing Step 24 above and its subsequent steps.

[0159] Step 29: If there are no untraversed video segments among the multiple video segments above, then according to the order of the multiple video segments, integrate the second video corresponding to the multiple video segments to obtain the body shape adjustment result corresponding to the long video above, so that the body shape adjustment result corresponding to the long video can represent the body shape adjustment result for each frame of the long video.

[0160] It should be noted that this application does not limit the implementation method of the above integration processing. For example, for any two video segments arranged adjacently in the above multiple video segments, when the video segment arranged earlier in these two video segments is regarded as the first sequence and the video segment arranged later in these two video segments is regarded as the second sequence, if the last R-frame image in the first sequence overlaps with the first R-frame image in the second sequence, the last R-frame image in the second video corresponding to the first sequence and the first R-frame image in the second video corresponding to the second sequence can be smoothed (such as by averaging) to obtain an R-frame smoothing result. This is so that the body shape adjustment result corresponding to the above long video can be generated based on the other images in the second video corresponding to the first sequence except for the last R-frame image, the R-frame smoothing result, and the other images in the second video corresponding to the second sequence except for the first R-frame image. This body shape adjustment result includes the other images in the second video corresponding to the first sequence except for the last R-frame image, the R-frame smoothing result, and the other images in the second video corresponding to the second sequence except for the first R-frame image.

[0161] Based on the relevant content of steps 21 to 29 above, in some scenarios, such as when the size of the input data of the timing module is fixed, if a user wants to perform body adjustment processing on a long video, the long video can be divided into multiple video segments, and then body adjustment processing can be performed on each video segment. Finally, the body adjustment results of these video segments can be integrated to obtain the body adjustment result corresponding to the long video. This can effectively meet the body adjustment processing of different video lengths, thereby helping to improve the body adjustment effect.

[0162] Furthermore, this application does not limit the training method of the target model mentioned above. For example, it can be implemented using any existing or future model training method, such as any diffusion model training method.

[0163] Furthermore, to better improve the model training effect, this application also provides a training method for the target model mentioned above. In this method, when the target model includes at least one temporal module, such as... Figure 4 The training process of the target model may include steps 31-32 below, where the sum (or residual) between the output data and the input data of any given time module is used as the input data of the next module corresponding to that time module.

[0164] Step 31: Set some or all parameters of each temporal module in the target model to zero, and train the other parts of the target model except for at least one temporal module based on the first image and the second image. The object described in the first image and the object described in the second image are the same object. This can achieve the first stage of training for the target model so that the trained target model can show better body shape adjustment performance on the image.

[0165] The first image refers to the image used in the first stage of training to provide information other than body shape for the image generation process.

[0166] The second image refers to the image used in the first stage of training to provide body shape information for the image generation process, so that the second image can be used to determine the guidance information for the image generation process, such as noise ground truth and image ground truth.

[0167] In addition, the step of “setting some or all of the parameters of each temporal module in the target model to zero” in step 31 above is used to ensure that these temporal modules do not work in the first stage of training, thereby ensuring that the other parts of the target model, except for at least one temporal module, can learn how to perform image-level body adjustment processing.

[0168] Furthermore, this application does not limit the implementation of the step of "setting some or all parameters of each temporal module in the target model to zero" mentioned above. For example, in some scenarios, when the temporal module includes a second conversion network, such as... Figure 4When the Project out module is shown, the specific step can be: set the parameters of the second transformation network of each temporal module in the target model to zero to ensure that the output data of each temporal module is 0, thereby ensuring that these temporal modules do not play a role in the first stage of training. This can achieve the first stage of training with as few setup operations as possible, thus effectively avoiding the defects that occur when relevant personnel perform a large number of setup operations, such as deleting temporal modules, to achieve the first stage of training, such as time overhead, manpower overhead, and a high probability of human error.

[0169] Furthermore, this application does not limit the implementation of step 31 above. For example, it may include steps 311-315 below.

[0170] Step 311: Set the parameters of the second transformation network of each temporal module in the target model to zero to ensure that the output data of each temporal module is 0, thereby ensuring that these temporal modules do not work in the first stage of training.

[0171] It should be noted that this application does not limit the execution time of step 311, but only needs to ensure that the execution time of step 311 is earlier than the execution time of step 314 below.

[0172] Step 312: Randomly select one frame from the sample video as the first image, and randomly select another frame from the sample video as the second image, so that the object described by the first image and the object described by the second image are the same object.

[0173] The sample video refers to the video used when training the target model; and this application does not limit the method of obtaining the sample video.

[0174] Step 313: After obtaining the noise randomly generated for the second image, the second image is denoised using the noise to obtain the denoised result (such as a noise image) corresponding to the second image, and the randomly generated noise is regarded as the label noise corresponding to the second image, so that the label noise is used to represent the noise actually added to the second image.

[0175] Step 314: The target model processes the image features of the first image, the semantic features of the first image, the state parameter prediction results (such as SMPL parameters) corresponding to the second image, and the noise addition results corresponding to the second image to obtain the prediction noise corresponding to the second image, so that the prediction noise can be used to represent the noise added for the prediction of the second image.

[0176] Step 315: Based on the difference between the predicted noise corresponding to the second image and the label noise corresponding to the second image, update the other parts of the target model except for at least one temporal module, and return to continue executing step 312 above and subsequent steps until the first preset stopping condition is reached.

[0177] The first preset stopping condition refers to the condition that must be met at the end of the first stage of training. Furthermore, this application does not limit the implementation of the first preset stopping condition. For example, the first preset stopping condition may include the model loss of the target model being lower than a preset first loss threshold. Alternatively, the first preset stopping condition may include the rate of change of the model loss of the target model being lower than a preset first rate of change threshold. Yet another example is that the first preset stopping condition may include the number of updates to other parts of the target model besides at least one temporal module reaching a preset first number threshold.

[0178] Furthermore, the model loss of the target model is used to describe the performance of the target model; and in the first training phase, the model loss of the target model can be determined based on the difference between the predicted noise corresponding to the second image and the label noise corresponding to the second image. It should be noted that this application does not limit the calculation method of the model loss; for example, it can be implemented using L2 loss.

[0179] Based on the relevant content of steps 311 to 315 above, for the first training stage, it is necessary to first set the parameters of the second transformation network of each temporal module in the target model to zero; then use some image pairs to train the other parts of the target model except for at least one temporal module, so that these parts can learn how to perform image-level body shape adjustment processing from these image pairs.

[0180] Research has shown that during the inference process, the target model is used to process multiple frames of images simultaneously. Therefore, in order to better meet this requirement, this application also provides a possible implementation of the first training stage mentioned above. In this method, when the target model is used to process F frames of images simultaneously, the first training stage can at least satisfy the following constraints: For each round of training in the first training stage, this round of training is implemented using F first images randomly extracted from the sample video and the second images corresponding to each first image, so that the target model can determine the prediction noise corresponding to each second image based on the F first images and the second images corresponding to each first image during this round of training.

[0181] It should be noted that this application does not limit the processing method of other parts of the target model other than at least one time-series module for F data (such as F first images, etc.). For example, the F data can be processed as a batch so that when the input data of these parts has a value of F at the size b, the purpose of processing the F data by these parts simultaneously (or in parallel) can be achieved.

[0182] Based on the relevant content of step 31 above, for the first stage of training of the target model, some or all parameters of each temporal module in the target model are first set to zero; then, some image pairs (such as the first image + the second image, etc.) are used to train the other parts of the target model except for at least one temporal module, so that the other parts of the target model except for at least one temporal module after the first stage of training can present better body shape adjustment performance on the image.

[0183] Step 32: Freeze the parameters of the target model except for at least one temporal module, and train at least one temporal module of the target model based on the first image sequence and the second image sequence. The object described by the first image sequence and the object described by the second image sequence are the same object. This enables the second stage of training for the target model so that the trained target model can present better body shape adjustment performance in the video.

[0184] The first image sequence refers to the sequence of images used in the second stage of training to provide information other than body shape for the video generation process. For example, the first image sequence may include images from frame k to frame (k+23) of the sample video, where k is a positive integer.

[0185] The second image sequence refers to the sequence of images used in the second stage of training to provide shape information for the video generation process. This second image sequence can then be used to determine guidance information for the video generation process, such as noise ground truth and image ground truth. For example, the second image sequence may include images from frame j to frame j+23 of the sample video, where j is a positive integer.

[0186] In addition, this application does not limit the implementation of step 32 above. For example, when the preset frame number threshold above is H, and the target model is used to process H images at the same time, where H is a positive integer, step 32 may specifically include steps 321-325 below.

[0187] Step 321: After completing the first training phase for the target model, freeze the parameters of all parts of the target model except for at least one temporal module to ensure that the parameters of these parts are not updated in the second training phase.

[0188] It should be noted that this application does not limit the execution time of step 321, but only needs to ensure that the execution time of step 321 is earlier than the execution time of step 324 below.

[0189] Step 322: Randomly generate k and j, such that neither k nor j is greater than the number of image frames in the sample video. Construct a first image sequence using the k-th frame to the (k+H-1)-th frame in the sample video, and construct a second image sequence using the j-th frame to the (j+H-1)-th frame in the sample video. Here, k and j are positive integers, and k and j can be the same or different; this application does not impose any restrictions on this.

[0190] Step 323: For any frame of the second image sequence, after obtaining the noise randomly generated for the image, the image is denoised using the noise to obtain the denoised result (such as a noise image) corresponding to the image, and the randomly generated noise is regarded as the label noise corresponding to the image, so that the label noise is used to represent the noise actually added to the image.

[0191] Step 324: The target model processes the image features of each frame in the first image sequence, the semantic features of each frame in the first image sequence, the state parameter prediction results (such as SMPL parameters) corresponding to each frame in the second image sequence, and the noise addition results corresponding to each frame in the second image sequence to obtain the prediction noise corresponding to each frame in the second image sequence, so that the prediction noise can be used to represent the noise added for the prediction of the corresponding image.

[0192] Step 325: Based on the difference between the predicted noise corresponding to each frame of the second image sequence and the label noise corresponding to each frame of the second image sequence, update at least one temporal module in the target model, and return to continue executing step 322 above and subsequent steps until the second preset stopping condition is reached.

[0193] The second preset stopping condition refers to the condition that must be met at the end of the second stage of training. Furthermore, this application does not limit the implementation of the second preset stopping condition. For example, the second preset stopping condition may include the model loss of the target model being lower than a preset second loss threshold. Alternatively, the second preset stopping condition may include the rate of change of the model loss of the target model being lower than a preset second rate of change threshold. Yet another example is that the second preset stopping condition may include the number of updates of at least one temporal module in the target model reaching a preset second number threshold.

[0194] Furthermore, the model loss of the target model is used to describe the performance of the target model; and in the second training phase, the model loss of the target model can be determined based on the difference between the predicted noise corresponding to each frame of the second image sequence and the label noise corresponding to each frame of the second image sequence. It should be noted that this application does not limit the calculation method of the model loss; for example, it can be implemented using L2 loss.

[0195] Based on the relevant content of steps 321 to 325 above, it can be seen that for the target model, after completing the first training stage, the other parts of the target model except for at least one temporal module have relatively good performance. Therefore, in order to avoid interference to the other parts in subsequent training, the other parts can be directly frozen to ensure that the parameters of the other parts are not updated in subsequent training. This ensures that in subsequent training, the at least one temporal module is guided to learn how to perform temporal consistency optimization through parameter updates, so that the finally trained target model can learn how to perform video-level body adjustment processing.

[0196] In addition, in order to further improve the model performance, this application also provides a possible implementation of step 32 above, in which step 32 may include at least steps 326-331 below.

[0197] Step 326: Obtain the label noise corresponding to each frame of the second image sequence.

[0198] It should be noted that this application does not limit the implementation of step 326 above. For example, it can be implemented by random generation.

[0199] Step 327: For any frame of the second image sequence, add noise to the image according to the label noise corresponding to the image to obtain the noise-added result (such as a noisy image).

[0200] Step 328: Based on the target model, the prediction results of the body shape parameters corresponding to each frame of the first image sequence, the second image sequence, and the noise addition results corresponding to each frame of the second image sequence, determine the prediction information corresponding to each frame of the second image sequence. The prediction information includes prediction noise and prediction image.

[0201] Specifically, for any frame in the second image sequence mentioned above, the prediction information corresponding to that image describes some information predicted for that image, such as noise prediction results and image generation results. Furthermore, the prediction information corresponding to that image may include the prediction noise corresponding to that image and the prediction image corresponding to that image. Specifically, the prediction noise corresponding to that image describes the noise added to the prediction of that image. The prediction image corresponding to that image describes the information carried by the image prediction.

[0202] Furthermore, this application does not limit the implementation of step 328 above. For example, in some scenarios, step 328 may specifically be: determining the prediction information corresponding to each frame of the second image sequence based on the target model, the first image sequence, the state parameter prediction results (such as SMPL parameters) corresponding to each frame of the second image sequence, and the noise addition results corresponding to each frame of the second image sequence.

[0203] Step 329: Determine the first loss based on the label noise corresponding to each frame of the second image sequence and the prediction noise corresponding to each frame of the second image sequence, so that the first loss can represent the performance of the target model.

[0204] It should be noted that this application does not limit the implementation of step 329 above; for example, it can be implemented using L2 loss.

[0205] Step 330: Based on the region segmentation results of each frame in the second image sequence, the object location detection results of each frame in the second image sequence, and the region segmentation results of the prediction images corresponding to each frame in the second image sequence, determine the second loss so that the second loss can represent the prediction performance of the target model in terms of object state.

[0206] Specifically, for any frame in the second image sequence mentioned above, the region segmentation result is used to describe the location of various parts of the foreground in the image. This allows the region segmentation result to represent the state of the foreground in the image, such as its size and pose. This information can then be used as guidance to help the target model better optimize its predictive performance regarding the foreground state. It should be noted that this application does not limit the method of obtaining the region segmentation result; for example, it can be implemented using any image segmentation method, such as a foreground segmentation method.

[0207] Furthermore, for any frame in the second image sequence mentioned above, the object location detection result is used to describe the position of the foreground in the image, so that the object location detection result can describe the layout structure in the image. This allows the object location detection result to be used as guiding information to help the target model better optimize its predictive performance in the foreground state. It should be noted that this application does not limit the implementation method of the object location detection result; for example, it can be implemented using bounding boxes or foreground mask maps. Moreover, this application does not limit the method of obtaining the object location detection result; for example, it can be implemented using any object detection method.

[0208] Furthermore, for any frame in the second image sequence mentioned above, the region segmentation result of the corresponding predicted image is used to describe the location of various parts of the foreground in the predicted image. This allows the region segmentation result to represent the state of the foreground in the predicted image, such as its size and pose. Consequently, the region segmentation result can represent the foreground prediction information for the image, such as the prediction information for various parts of the image. This enables subsequent evaluation of the target model's prediction performance regarding object states based on the region segmentation result. It should be noted that this application does not limit the method of obtaining the region segmentation result; for example, it can be implemented using any image segmentation method, such as a foreground segmentation method.

[0209] Furthermore, this application does not limit the implementation of step 330 above. For example, it can specifically be: for any frame of the second image sequence above, first calculate the difference (such as distance) between the region segmentation result of the image and the region segmentation result of the corresponding predicted image, so that the difference can represent the difference between the two images in the entire image range; then multiply the difference with the object position detection result (such as the foreground mask map) of the image to obtain the state prediction loss corresponding to the image, so that the state prediction loss can represent the difference between the two images in the foreground range, so that the second loss can be determined based on the state prediction loss corresponding to all images in the second image sequence, so that the second loss can represent the prediction performance of the target model in terms of object state.

[0210] Step 331: Update at least one temporal module in the target model based on the sum of the first loss and the second loss.

[0211] Based on the relevant content of steps 326 to 331 above, it can be seen that in the second training stage, not only can the loss between the predicted noise and the label noise be calculated, but it is also necessary to further calculate the loss between the object state described by each frame of the second image sequence and the object state described by the corresponding predicted image. This is to ensure that the object state presented in the predicted image generated by the target model is as consistent as possible with the object state presented in the ground truth image corresponding to the predicted image. This allows the target model to better learn how to predict the object state in the second training stage, thereby improving the model training effect.

[0212] In addition, in order to further improve the model performance, this application also provides a possible implementation of step 32 above, in which step 32 may include at least steps 332-335 below.

[0213] Step 332: Obtain the first image sequence, the second image sequence, and the label information corresponding to each frame of the second image sequence.

[0214] For any frame in the second image sequence above, the label information corresponding to the image refers to the ground truth information associated with the image, such as the ground truth of noise and the ground truth of the image, so that the label information can be used as guidance information to guide the target model to generate and process the image.

[0215] Furthermore, this application does not limit the implementation of the above-mentioned label information. For example, in some scenarios, for any frame of the second image sequence above, the label information corresponding to the image may include the label noise corresponding to the image, so that the label information can describe the noise actually added to the image.

[0216] For example, in some scenarios, for any frame of the second image sequence mentioned above, the label information corresponding to the image can include the label noise corresponding to the image and the image itself, so that the label information can not only describe the noise actually added to the image, but also represent the true value of the image obtained by denoising the noise added to the image.

[0217] As can be seen, in one possible implementation, for any frame of the second image sequence above, the process of obtaining the label information corresponding to the image can be as follows: first, the image is directly regarded as the label image corresponding to the image, so that the label image can represent the information actually carried by the image, thereby enabling the label image to represent the true value of the image obtained by denoising the noise result corresponding to the image; then, based on the label image corresponding to the image and the label noise corresponding to the image, the label information corresponding to the image is determined, so that the label information includes the label image and the label noise.

[0218] Step 333: Perform masking processing on at least one frame of the first image sequence to obtain a processed sequence. The processed sequence includes the masking processing result of the at least one frame of the image. For any frame of the at least one image, the information described by the masking processing result of the image is less than the information described by the image itself.

[0219] The masking process is used to remove some or all of the information in an image; and this application does not limit the masking process. For example, it can specifically be: adjusting some or all of the pixel values ​​in the image to preset pixel values, such as pixel values ​​used to represent black.

[0220] Based on the relevant content of step 333 above, for some scenarios, after obtaining the first image sequence, one or more frames can be randomly selected from the first image sequence for masking, such as replacing them with completely black images, to obtain a processed sequence. This results in some images in the processed sequence that have lost some or all of their information, so that these images that have lost some or all of their information cannot provide sufficient image information for the corresponding image generation process. This allows the target model to better learn how to perform image generation processing based on other information (such as adjacent images, object state, etc.). This can effectively prevent the model from only learning reference images (such as each frame in the first image sequence) and ignoring other information (such as SMPL parameters) when the training model is overfitted.

[0221] Step 334: Based on the target model, the processed sequence, and the prediction results of the body shape parameters corresponding to each frame of the second image sequence, determine the prediction information corresponding to each frame of the second image sequence.

[0222] It should be noted that this application does not limit the implementation of step 334 above. For example, the implementation of step 334 is similar to the implementation of step 328 above, and will not be repeated here for the sake of brevity.

[0223] Step 335: Update at least one temporal module in the target model based on the prediction information corresponding to each frame of the second image sequence and the label information corresponding to each frame of the second image sequence.

[0224] Based on the relevant content of steps 332 to 335 above, for the second training stage, after obtaining the first image sequence, random masking can be performed on the first image sequence to replace some images in the first image sequence with completely black images. This helps to increase the perturbation of the data, thereby effectively preventing the model from only learning the reference image (such as each frame in the first image sequence) and ignoring other information guidance when the training model is overfitted, thus helping to improve the training effect of the model.

[0225] Based on the relevant content of steps 31 to 32 above, it can be seen that in some scenarios, the training process of the target model can be completed by using a two-stage training method, so that the trained target model has better video generation performance, thereby making the video generated by the target model present better effects, such as temporal consistency, which is conducive to improving the body shape adjustment effect.

[0226] Based on the video generation method provided in the embodiments of this application, the embodiments of this application also provide a video generation apparatus, which will be discussed below. Figure 5 Explanation and clarification will be provided. Among them, Figure 5 This is a schematic diagram of a video generation apparatus provided in an embodiment of this application. It should be noted that for technical details of the video generation apparatus provided in this application embodiment, please refer to the relevant content of the video generation method described above.

[0227] like Figure 5 As shown, the video generation apparatus 500 provided in this application embodiment includes:

[0228] The data acquisition unit 501 is used to acquire a first video, wherein each frame of the first video includes the target object.

[0229] The parameter prediction unit 502 is used to perform body shape parameter prediction processing on any frame of the first video to obtain the body shape parameter prediction result corresponding to the image. The body shape parameter prediction result is used to describe the body shape of the target object in the image.

[0230] The parameter adjustment unit 503 is used to adjust the predicted body shape parameters of any frame in the first video according to the body shape adjustment information specified for the target object, so as to obtain the body shape parameter adjustment result corresponding to the image.

[0231] The video generation unit 504 is used to generate a second video based on the first video and the body shape parameter adjustment results corresponding to each frame image in the first video. The i-th frame image in the second video is used to represent the body shape adjustment result for the i-th frame image in the first video. The i-th frame image in the second video is generated based on multiple frames of images arranged sequentially in the first video and the body shape parameter adjustment results corresponding to the multiple frames of images. The distance between the arrangement position of each frame image in the multiple frames of images in the first video and the arrangement position of the i-th frame image in the first video is not greater than a preset distance threshold. i is a positive integer, and i ≤ the number of image frames in the second video or the number of image frames in the first video.

[0232] In one possible implementation, the body shape described by the i-th frame image in the second video is different from the body shape described by the i-th frame image in the first video, and the other information described by the i-th frame image in the second video, except for the body shape, is consistent with the other information described by the i-th frame image in the first video, except for the body shape.

[0233] In one possible implementation, the predicted body shape parameters for each frame of the second video remain consistent.

[0234] In one possible implementation, the second video is generated using a target model; the target model includes at least one processing unit, the processing unit including an image-level processing module and a temporal module, the image-level processing module being used to implement individual processing procedures for each frame of the multi-frame images, and the temporal module being used to perform attention processing on the execution results of the individual processing procedures for the multi-frame images in the temporal direction.

[0235] In one possible implementation, the at least one processing unit includes a plurality of upsampling modules, a timing module corresponding to each of the upsampling modules, a plurality of downsampling modules, and a timing module corresponding to each of the downsampling modules; for any one of the upsampling modules, the timing module corresponding to the upsampling module is used to perform attention processing on the output data of the upsampling module in the time direction; for any one of the downsampling modules, the timing module corresponding to the downsampling module is used to perform attention processing on the output data of the downsampling module in the time direction.

[0236] In one possible implementation, the attention processing is further implemented based on the temporal information of each frame in the multi-frame images; for any frame in the multi-frame images, the temporal information of the image is used to describe the position of the image in the first video.

[0237] In one possible implementation, the timing module includes an integration network, a first transformation network, at least one self-attention network, a second transformation network, and a splitting network. The integration network integrates the input data of the timing module to obtain an integration result, the size of which differs from the size of the input data of the timing module. The input data of the timing module includes the execution results of individual processing procedures for the multi-frame images. The first transformation network converts the integration result from a first representation to a second representation to obtain a first transformation result, the first representation describing image features and the second representation describing video features. The at least one self-attention network performs self-attention processing on the first transformation result to obtain a self-attention processing result. The second transformation network converts the self-attention processing result from the second representation to the first representation to obtain a second transformation result. The splitting network splits the second transformation result to obtain a splitting result, the size of which is the same as the size of the input data of the timing module.

[0238] In one possible implementation, the second video is generated using a target model; the target model includes at least one temporal module.

[0239] The training process of the target model includes: setting some or all parameters of each temporal module in the target model to zero, and training the other parts of the target model except for the at least one temporal module based on the first image and the second image, wherein the object described by the first image and the object described by the second image are the same object; freezing the parameters of the other parts of the target model except for the at least one temporal module, and training at least one temporal module in the target model based on the first image sequence and the second image sequence, wherein the object described by the first image sequence and the object described by the second image sequence are the same object.

[0240] In one possible implementation, the temporal module includes a second transformation network; the training process of the target model specifically includes: setting the parameters of the second transformation network of each temporal module in the target model to zero.

[0241] In one possible implementation, the training process of the at least one temporal module includes: acquiring the first image sequence, the second image sequence, and label information corresponding to each frame image in the second image sequence; performing masking processing on at least one frame image in the first image sequence to obtain a processed sequence, the processed sequence including the masking processing result of the at least one frame image, wherein for any frame image in the at least one frame image, the information described by the masking processing result of that image is less than the information described by that image; determining the prediction information corresponding to each frame image in the second image sequence based on the target model, the processed sequence, and the body shape parameter prediction results corresponding to each frame image in the second image sequence; and updating at least one temporal module in the target model based on the prediction information corresponding to each frame image in the second image sequence and the label information corresponding to each frame image in the second image sequence.

[0242] In one possible implementation, the training process of the at least one temporal module includes: acquiring the label noise corresponding to each frame image in the second image sequence; for any frame image in the second image sequence, adding noise to the image according to the label noise corresponding to that image to obtain the noise-added result corresponding to that image; determining the prediction information corresponding to each frame image in the second image sequence based on the target model, the first image sequence, the body shape parameter prediction results corresponding to each frame image in the second image sequence, and the noise-added results corresponding to each frame image in the second image sequence, the prediction information including prediction noise and prediction image; determining a first loss based on the label noise corresponding to each frame image in the second image sequence and the prediction noise corresponding to each frame image in the second image sequence; determining a second loss based on the region segmentation results of each frame image in the second image sequence, the object position detection results of each frame image in the second image sequence, and the region segmentation results of the prediction image corresponding to each frame image in the second image sequence; and updating at least one temporal module in the target model based on the sum of the first loss and the second loss.

[0243] Based on the aforementioned content of the video generation device 500, the working principle of the video generation device 500 provided in this application is as follows: First, a first video is acquired so that each frame of the first video includes the target object; second, for any frame of the first video, body shape parameter prediction processing is performed on the image to obtain the corresponding body shape parameter prediction result, so that the body shape parameter prediction result is used to describe the body shape of the target object in the image; then, for any frame of the first video, based on the body shape adjustment information specified for the target object (such as losing 20% ​​or other information), the corresponding body shape parameter prediction result is adjusted to obtain the corresponding body shape parameter adjustment result; finally, based on the first video and the corresponding body shape parameter adjustment results of each frame of the first video, a second video is generated so that the second video is used to represent the body shape adjustment result for the first video, such that the i-th frame of the second video is used to represent the body shape adjustment result for the i-th frame of the first video, where i is a positive integer and i ≤ the number of image frames in the second video or the number of image frames in the first video. In this process, the i-th frame image in the second video is generated based on multiple frames arranged sequentially in the first video and the body shape parameter adjustment results corresponding to those multiple frames. Furthermore, the distance between the arrangement position of each frame image in the first video and the arrangement position of the i-th frame image in the first video is not greater than a preset distance threshold. This ensures that the i-th frame image in the second video not only satisfies the body shape constraints described by the body shape parameter adjustment results corresponding to the i-th frame image in the first video, but also satisfies other constraints described by the multiple frames besides body shape, such as the temporal dependencies of some information. As a result, the second video has better temporal consistency, which is beneficial for improving the body shape adjustment effect.

[0244] In addition, this application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory so that the electronic device performs any implementation of the video generation method provided in this application.

[0245] See Figure 6 This diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0246] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0247] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0248] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0249] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0250] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the video generation method provided in this application.

[0251] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0252] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0253] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0254] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0255] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0256] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0257] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0258] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0259] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0260] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0261] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0262] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0263] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0264] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of video generation, the method comprising: The method comprises: acquiring a first video, each frame image of the first video comprising a target object; for any frame image of the first video, performing body shape parameter prediction processing on the image to obtain a body shape parameter prediction result corresponding to the image, the body shape parameter prediction result being used to describe the body shape of the target object in the image; for any frame image of the first video, performing adjustment processing on the body shape parameter prediction result corresponding to the image according to body shape adjustment information specified for the target object to obtain a body shape parameter adjustment result corresponding to the image; generating a second video according to the first video and the body shape parameter adjustment results corresponding to the frame images of the first video, the i-th frame image of the second video being used to represent the body shape adjustment result of the i-th frame image of the first video, the i-th frame image of the second video being generated according to a plurality of frame images arranged in sequence in the first video and the body shape parameter adjustment results corresponding to the plurality of frame images, the distance between the arrangement position of each frame image in the plurality of frame images in the first video and the arrangement position of the i-th frame image of the first video in the first video being not greater than a preset distance threshold, i being a positive integer, i≤the number of frame images in the second video or the number of frame images in the first video.

2. The method of claim 1, wherein, The body shape described by the i-th frame image of the second video is different from the body shape described by the i-th frame image of the first video, and the information other than the body shape described by the i-th frame image of the second video is consistent with the information other than the body shape described by the i-th frame image of the first video; and / or, the body shape parameter prediction results corresponding to the frame images of the second video are consistent.

3. The method of claim 1, wherein, The second video is generated by using a target model; The target model comprises at least one processing unit, the processing unit comprising an image-level processing module and a timing module, the image-level processing module being used to respectively implement a separate processing process for each frame image of the plurality of frame images, and the timing module being used to perform attention processing on the execution results of the separate processing processes for the plurality of frame images in the time direction.

4. The method of claim 3, wherein, The at least one processing unit comprises a plurality of up-sampling modules, a timing module corresponding to each up-sampling module, a plurality of down-sampling modules, and a timing module corresponding to each down-sampling module; for any up-sampling module, the timing module corresponding to the up-sampling module is used to perform attention processing on the output data of the up-sampling module in the time direction; for any down-sampling module, the timing module corresponding to the down-sampling module is used to perform attention processing on the output data of the down-sampling module in the time direction.

5. The method of claim 3, wherein, The attention processing is further implemented according to the time information of each frame image of the plurality of frame images; for any frame image of the plurality of frame images, the time information of the image is used to describe the arrangement position of the image in the first video.

6. The method according to any one of claims 3-5, characterized in that, The timing module comprises an integration network, a first conversion network, at least one self-attention network, a second conversion network, and a splitting network; The integration network is configured to perform integration processing on input data of the time sequence module to obtain an integration result, wherein a size of the integration result is different from a size of the input data of the time sequence module, and the input data of the time sequence module comprises execution results of separate processing processes for the multiple frames of images. The first conversion network is configured to convert the integration result from a first expression mode to a second expression mode to obtain a first conversion result, wherein the first expression mode is used to describe image features, and the second expression mode is used to describe video features. The at least one self-attention network is configured to perform self-attention processing on the first conversion result to obtain a self-attention processing result. The second conversion network is configured to convert the self-attention processing result from the second expression mode to the first expression mode to obtain a second conversion result. The split network is configured to perform split processing on the second conversion result to obtain a split result, wherein a size of the split result is the same as the size of the input data of the time sequence module.

7. The method of claim 1, wherein, The second video is generated by using a target model. The target model comprises at least one time sequence module. The training process of the target model comprises: setting part or all parameters of each time sequence module in the target model to zero, and training other parts of the target model except the at least one time sequence module according to a first image and a second image, wherein an object described by the first image and an object described by the second image are the same object; freezing parameters of other parts of the target model except the at least one time sequence module, and training the at least one time sequence module in the target model according to a first image sequence and a second image sequence, wherein an object described by the first image sequence and an object described by the second image sequence are the same object.

8. The method of claim 7, wherein, The time sequence module comprises a second conversion network. The setting part or all parameters of each time sequence module in the target model to zero comprises: setting parameters of the second conversion network of each time sequence module in the target model to zero.

9. The method of claim 7, wherein, The training process of the at least one time sequence module comprises: obtaining the first image sequence, the second image sequence, and label information corresponding to each frame of image in the second image sequence; performing mask processing on at least one frame of image in the first image sequence to obtain a processed sequence, wherein the processed sequence comprises a mask processing result of the at least one frame of image, and for any one frame of image in the at least one frame of image, the mask processing result of the image describes less information than the image; determining prediction information corresponding to each frame of image in the second image sequence according to the target model, the processed sequence, and a body parameter prediction result corresponding to each frame of image in the second image sequence; updating the at least one time sequence module in the target model according to the prediction information corresponding to each frame of image in the second image sequence and label information corresponding to each frame of image in the second image sequence.

10. The method according to any one of claims 7-9, characterized in that, The training process of the at least one time sequence module comprises: obtaining label noise corresponding to each frame of image in the second image sequence; For any one image in the second image sequence, the image is subjected to noise adding processing according to the label noise corresponding to the image, to obtain a noise adding result corresponding to the image; According to the target model, the first image sequence, the body shape parameter prediction result corresponding to each image in the second image sequence, and the noise adding result corresponding to each image in the second image sequence, the prediction information corresponding to each image in the second image sequence is determined, and the prediction information includes predicted noise and a predicted image; According to the label noise corresponding to each image in the second image sequence and the predicted noise corresponding to each image in the second image sequence, a first loss is determined; According to the region segmentation result of each image in the second image sequence, the object position detection result of each image in the second image sequence, and the region segmentation result of the predicted image corresponding to each image in the second image sequence, a second loss is determined; According to the sum value between the first loss and the second loss, at least one time sequence module in the target model is updated.

11. A video generating apparatus characterized by comprising: Comprise: A data acquisition unit is configured to acquire a first video, each frame of image in the first video comprising a target object; A parameter prediction unit is configured to, for any one image in the first video, perform body shape parameter prediction processing on the image to obtain a body shape parameter prediction result corresponding to the image, the body shape parameter prediction result being used to describe the body shape of the target object in the image; A parameter adjustment unit is configured to, for any one image in the first video, perform adjustment processing on the body shape parameter prediction result corresponding to the image according to the body shape adjustment information specified for the target object to obtain a body shape parameter adjustment result corresponding to the image; A video generation unit is configured to generate a second video according to the first video and the body shape parameter adjustment result corresponding to each image in the first video, the i-th image in the second video being used to represent the body shape adjustment result of the i-th image in the first video, the i-th image in the second video being generated according to a plurality of images arranged in sequence in the first video and the body shape parameter adjustment result corresponding to the plurality of images, the distance between the arrangement position of each image in the plurality of images in the first video and the arrangement position of the i-th image in the first video in the first video being not greater than a preset distance threshold, i being a positive integer, i≤the number of image frames in the second video or the number of image frames in the first video.

12. An electronic device, comprising: The device comprises a processor and a memory; The memory is configured to store instructions or computer programs; The processor is configured to execute the instructions or computer programs in the memory to enable the electronic device to perform the method of any one of claims 1-10.

13. A computer readable medium characterized by The computer readable medium stores instructions or computer programs, when the instructions or computer programs are run on a device, enabling the device to perform the method of any one of claims 1-10.

14. A computer program product, characterised in that, It comprises a computer program carried on a non-transitory computer readable medium, the computer program containing program code for carrying out the method of any one of claims 1-10.

Images