Model training method and device, information determination method and device, equipment, storage medium and computer program product

Abstract

The embodiment of the invention discloses a model training method. The method comprises the following steps: acquiring a sample picture for a sample object and a sample background video for a background; wherein the sample picture is a picture without a background; generating a sample combined video based on the sample picture and the sample background video; and based on the sample background video and the sample combination video, performing model training on an initial video generation model to obtain a target video generation model. The embodiment of the invention further discloses an information determination method and device, equipment, a storage medium and a computer program product.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a model training, information determination method, device, equipment, storage medium and computer program product. Background Art

[0002] As image generation technology continues to mature, technologies such as text-generated images and image style transfer have been able to meet people's demand for high-quality images. In the field of e-commerce advertising, image generation technology has been able to generate artificial intelligence (AI) product background images with various background templates for a white background image of a product. However, the image itself lacks time dimension information and cannot express the movement state and law of the entity, which makes the AI ​​product background image express limited information; based on the fact that videos can show the multi-angle details of the product and enhance the user's visual and auditory experience through dynamic effects and sound, videos have been widely favored in the field of e-commerce advertising. At present, video generation technology mainly targets the fields of general text-generated videos and image-generated videos. These models have the problem of not being able to maintain the details of the product when generating e-commerce videos, and the background transformation cannot achieve coherent and regular movement, which leads to poor visual effects. Summary of the invention

[0003] In order to solve the above-mentioned technical problems, the embodiments of the present application hope to provide a model training, information determination method, device, equipment, storage medium and computer program product, which solves the problem that the video generated for the product in the related technology cannot maintain the details of the product and the background change cannot achieve a coherent and regular movement, thereby improving the visual effect.

[0004] The technical solution of this application is implemented as follows:

[0005] A model training method, the method comprising:

[0006] Obtaining a sample picture for a sample object and a sample background video for a background; wherein the sample picture is a picture without a background;

[0007] Generate a sample combined video based on the sample picture and the sample background video;

[0008] Based on the sample background video and the sample combined video, the initial video generation model is trained to obtain a target video generation model.

[0009] In the above solution, the step of obtaining a sample image for a sample object includes:

[0010] Get an initial image for the sample object;

[0011] The target cutout model is used to process the initial image to obtain the sample image.

[0012] In the above solution, generating a sample combined video based on the sample picture and the sample background video includes:

[0013] For each frame of the sample background video, dividing each frame of the sample background video into a plurality of sub-images;

[0014] For each frame of the picture, generating a combined picture based on any sub-picture of the multiple sub-pictures and any picture of the sample pictures;

[0015] The sample combined video is generated based on a plurality of component graphs.

[0016] In the above solution, the initial video generation model is trained based on the sample background video and the sample combined video to obtain the target video generation model, including:

[0017] Extract features from each frame of the sample background video to obtain a sample feature map;

[0018] Using the first sub-initial video generation model to process the sample feature map to obtain a video to be processed;

[0019] Based on the video to be processed and the sample combined video, the second sub-initial video generation model is trained to obtain the target video generation model; wherein the initial video generation model includes the first sub-initial video generation model and the second sub-initial video generation model.

[0020] In the above scheme, the training of the second sub-initial video generation model based on the video to be processed and the sample combined video to obtain the target video generation model includes:

[0021] Determining a target coefficient, and determining a target loss function based on the target coefficient and a basic loss function;

[0022] Based on the video to be processed, the sample combination video and the target loss function, the second sub-initial video generation model is trained to obtain the target video generation model.

[0023] In the above scheme, the determination of the target coefficient includes:

[0024] Determine a first pixel value of a first frame image and a second pixel value of a last frame image of the sample combined video;

[0025] A difference between the first pixel value and the second pixel value is calculated, and the target coefficient is determined based on the difference.

[0026] A method for determining information, the method comprising:

[0027] Get the image to be processed for the target object;

[0028] Using a target video generation model to process the image to be processed to obtain a target video for the target object;

[0029] The target video generation model may be obtained by training using the above-mentioned model training method.

[0030] A model training device, comprising:

[0031] A first acquisition unit is used to acquire a sample picture for a sample object and a sample background video for a background; wherein the sample picture is a picture without a background;

[0032] A first processing unit, configured to generate a sample combined video based on the sample picture and the sample background video;

[0033] The training unit is used to perform model training on the initial video generation model based on the sample background video and the sample combination video to obtain a target video generation model.

[0034] An information determination device, the device comprising:

[0035] A second acquisition unit is used to acquire a to-be-processed image of a target object;

[0036] A second processing unit is used to process the image to be processed using a target video generation model to obtain a target video for the target object;

[0037] The target video generation model may be obtained by training using the above-mentioned model training method.

[0038] A model training device, the device comprising: a first processor, a first memory and a first communication bus;

[0039] The first communication bus is used to realize the communication connection between the first processor and the first memory;

[0040] The first processor is used to execute the model training program in the first memory to implement the steps of the above-mentioned model training method.

[0041] An information determination device, the device comprising: a second processor, a second memory, and a second communication bus;

[0042] The second communication bus is used to realize the communication connection between the second processor and the second memory;

[0043] The second processor is used to execute the information determination program in the second memory to implement the steps of the above-mentioned information determination method.

[0044] A computer-readable storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the steps of the above-mentioned model training method or information determination method.

[0045] A computer program product comprises a computer program, wherein the computer program implements the above method when executed by a processor.

[0046] Because it is possible to obtain sample images without background and sample background videos for the sample objects, generate sample combination videos based on the sample images and the sample background videos, and train the initial video generation model based on the sample background videos and the sample combination videos to obtain the target video generation model, in this way, the target video generation model can be obtained by training the sample combination videos generated according to the sample images without background and the sample background videos corresponding to the background, that is, the sample background videos are used when generating the target video generation model, thereby fully ensuring the motion information of the background, and the sample combination videos generated according to the sample images and the sample background videos are also used, thereby ensuring the diversity of data during training, thereby solving the problem that the videos generated for the products in the related technologies cannot maintain the details of the products and the background changes cannot achieve coherent and regular motion, thereby improving the visual effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of generating a sample combination video in a model training method provided in an embodiment of the present application;

[0049] Figure 3 A flowchart of another model training method provided in an embodiment of the present application;

[0050] Figure 4 A flowchart of an information determination method provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of an information determination device provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of the structure of an information determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0056] It should be understood that the "embodiments of the present application" or "the aforementioned embodiments" mentioned throughout the specification mean that specific features, structures or characteristics related to the embodiments are included in at least one embodiment of the present application. Therefore, "in the embodiments of the present application" or "in the aforementioned embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0057] Unless otherwise specified, the electronic device performs any step in the embodiments of the present application, and the processor of the electronic device may perform the step. It is also worth noting that the embodiments of the present application do not limit the order in which the electronic device performs the following steps. In addition, the methods used to process data in different embodiments may be the same method or different methods. It should also be noted that any step in the embodiments of the present application can be independently executed by the electronic device, that is, when the electronic device executes any step in the following embodiments, it may not rely on the execution of other steps.

[0058] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The present application embodiment provides a model training method, which can be applied to a model training device, referring to Figure 1 As shown, the method may include the following steps:

[0060] Step 101: Obtain a sample image for a sample object and a sample background video for the background.

[0061] The sample image is an image without a background.

[0062] In the embodiments of the present application, the sample objects may refer to various commodities; the sample images may be a large number of commodity perspective images without background obtained about various commodities; the sample background videos may refer to a large number of background videos obtained about various backgrounds. It should be noted that the sample images may be used as the header image library of the commodity. product , the sample background video can be used as the background video library V bg .

[0063] Specifically, the model training device can obtain sample background videos based on background keywords; in a feasible implementation, such as Figure 2 As shown, you can select and download sample background videos from open source, copyright-free video websites by searching for background keywords such as "natural background", "rain and snow", and "starry sky".

[0064] Step 102: Generate a sample combined video based on the sample image and the sample background video.

[0065] In the embodiment of the present application, the model training device can first divide the pictures in the sample background video, and then generate a sample combination video based on the divided pictures and the sample pictures. It should be noted that the sample combination video is a large number of videos that meet the requirements of fixed products and naturally changing backgrounds; and the sample combination video can be used with V product To express.

[0066] Step 103: Based on the sample background video and the sample combination video, the initial video generation model is trained to obtain a target video generation model.

[0067] In the embodiment of the present application, the sample background video and the sample combination video can be used as input parameters of the initial video generation model, that is, the sample background video and the sample combination video can be input into the initial video generation model for processing to implement model training of the initial video generation model, thereby obtaining a target video generation model. The target video generation model is a model for generating videos about products.

[0068] It should be noted that if Figure 3 The model training device shown may first perform a first training on the initial video generation model based on the sample background video (i.e., training stage 1), and then continue to perform a second training on the initial video generation model based on the sample combination video and the video to be processed obtained after the first training (i.e., training stage 2) to obtain a target video generation model.

[0069] In other embodiments of the present application, the “obtaining a sample image for a sample object” in the above step 101 can be implemented in the following manner:

[0070] A1. Obtain an initial image of the sample object.

[0071] The initial image may refer to the product images obtained for various products. Figure 2 The initial image shown may be downloaded from an e-commerce website.

[0072] A2. Use the target cutout model to process the initial image to obtain a sample image.

[0073] The target cutout model may refer to any cutout model that can perform cutout processing on an image; specifically, Figure 2 As shown, the target cutout model can be used to cut out the initial image to obtain an image having only the sample object (ie, the sample image).

[0074] In other embodiments of the present application, the above step 102 may be implemented in the following manner:

[0075] B1. For each frame of the sample background video, divide each frame into multiple sub-images;

[0076] Among them, the model training device can be obtained from the background video library V bg A background video is randomly sampled, and each frame of the background video is evenly divided to obtain multiple sub-images; in a feasible implementation method, each frame of the background video can be evenly divided into 9 sub-images in a nine-square grid manner.

[0077] It should be noted that the background video library V bg Each frame of each video in can be divided using this method.

[0078] B2. For each frame of the picture, generate a combined picture based on any sub-picture in the multiple sub-pictures and any picture in the sample pictures;

[0079] Specifically, Figure 2 As shown in the figure, for each frame of each background video, you can product A product perspective image is randomly sampled, and the sampled product perspective image is combined with any sub-image in multiple sub-images to obtain a combined image; it should be noted that each background video can correspond to multiple combined images; and the background video library V bg Each video in the product perspective library I can use this method product Any of the images in the .

[0080] B3. Generate sample combination videos based on multiple component graphs.

[0081] Among them, for the background video library V bgFor each video in the sample, the multiple combination images generated corresponding to each video can be combined according to the sequence of video frames to generate a combined video; it should be noted that the sample combined video V product A large number of combined videos can be included.

[0082] In other embodiments of the present application, the above step 103 can be implemented in the following manner:

[0083] C1. Extract features from each frame of the sample background video to obtain a sample feature map.

[0084] C2. Use the first sub-initial video generation model to process the sample feature map to obtain the video to be processed.

[0085] The sample feature map may be input into the first sub-initial video generation model, and then the first sub-initial video generation model processes the sample feature map to obtain the video to be processed.

[0086] C3. Based on the video to be processed and the sample combination video, the second sub-initial video generation model is trained to obtain a target video generation model.

[0087] The initial video generation model includes a first sub-initial video generation model and a second sub-initial video generation model; it should be noted that the first sub-initial video generation model and the second sub-initial video generation model may be the same.

[0088] Specifically, the first sub-initial video generation model and the second sub-initial video generation model can be diffusion models; in a feasible implementation, the first sub-initial video generation model and the second sub-initial video generation model can be latent diffusion models (Latent Diffusion Models, LDMs). Specifically, given a video x∈R L×3×H×W , first use the encoder of the variational auto encoder (VAE) model to encode it frame by frame into a potential representation z = Enc (x), z ∈ R L×C×h×w Then, the forward diffusion process z is carried out in this latent space t =q(z0,t) and the reverse denoising process z t =p θ (z t-1 ,c,t), where c represents the denoising condition, which is the conditional image (i.e., the first frame of the input video) in this paper. Finally, the generated video is obtained by the decoder x=Dec(z) of the VAE model. In this application, the model training is based on the open source video generation model Dynamicrafter.

[0089] Specifically, the video to be processed and the sample combination video can be input into the second sub-initial video generation model, and the second sub-initial video generation model can be trained using the video to be processed and the sample combination video, and finally a target video generation model can be obtained.

[0090] It should be noted that in this application, the background motion is first fully trained based on the sample background video, and the model only learns the background video library V bg A large number of pure background videos are used to learn the laws of natural motion and simulate the motion state of the physical world; then, the model learns to maintain the details of the product entity, and uses the sample combination video and the video to be processed to further fine-tune the model so that the model learns how to maintain the details of the product; in this way, the training model can maintain the natural changes of the background while maintaining the physical details of the product to achieve the purpose of promotion.

[0091] In other embodiments of the present application, the initial video generation model is trained based on the video to be processed and the sample combination video to obtain the target video generation model, including:

[0092] Determine a target coefficient, and determine a target loss function based on the target coefficient and the basic loss function;

[0093] The target coefficient may be determined based on the pixel values ​​of the image of the sample combination video. Specifically, it may be determined based on the pixel values ​​of the first frame image and the last frame image of the sample combination video. It should be noted that the target loss function may refer to a dynamic area loss function.

[0094] It should be noted that the formula To determine the target loss function; where E t represents the basic loss function, Represents the target coefficient; it should be noted that the value of τ can be 1.

[0095] Based on the video to be processed, the sample combination video and the target loss function, the second sub-initial video generation model is trained to obtain the target video generation model.

[0096] Specifically, the video to be processed and the sample combination video can be input into the second sub-initial video generation model, and the second sub-initial video generation model can be used to process the video to be processed and the sample combination video to generate a video. Then, the target loss function can be used to compare the generated video and the sample combination video, and the parameters of the first sub-initial video generation model and the second sub-initial video generation model can be updated according to the comparison results. Thereafter, the model can continue to be trained based on the sample background video and the sample combination video until the result determined by the target loss function meets the requirements, and then the target video generation model can be trained.

[0097] It should be noted that by introducing the target loss function, the learning weight is increased in the dynamic area, which can significantly improve the conditional image leakage problem and optimize the video motion amplitude, thereby making the video generation effect of the target video generation model better.

[0098] In other embodiments of the present application, the above-mentioned determination of the target coefficient includes:

[0099] Determine a first pixel value of a first frame image and a second pixel value of a last frame image of a sample combined video;

[0100] A difference between the first pixel value and the second pixel value is calculated, and a target coefficient is determined based on the difference.

[0101] The first pixel value can be represented by c, and the second pixel value can be represented by x. f It means that after calculating the difference, the absolute value of the difference can be taken, and then the absolute value of the difference can be processed to obtain the target coefficient; specifically, the formula can be used Calculate the target coefficient. It should be noted that the larger the absolute value of the pixel difference, the greater the motion change in the area, and the model needs to focus on learning, and then the model is trained based on the target loss function to optimize the video motion amplitude.

[0102] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0103] The model training method provided in the embodiments of the present application can use a sample combination video generated from a sample image without a background and a sample background video, and a sample background video corresponding to the background to train a target video generation model. That is, the sample background video is used when generating the target video generation model to fully ensure the motion information of the background, and the sample combination video generated from the sample image and the sample background video is also used to ensure the diversity of data during training, thereby solving the problem that the videos generated for the products in the related technology cannot maintain the details of the products and the background changes cannot achieve coherent and regular motion, thereby improving the visual effect.

[0104] Based on the above embodiments, the embodiments of the present application provide an information determination method, referring to Figure 4 As shown, the method may include the following steps:

[0105] Step 201: The information determination device obtains a to-be-processed picture for a target object.

[0106] The target object may be any product for which a corresponding video needs to be generated; and the image to be processed may be an image of the product.

[0107] Step 202: The information determination device uses a target video generation model to process the image to be processed to obtain a target video for the target object.

[0108] In the embodiment of the present application, the image to be processed can be used as an input parameter, that is, the image to be processed is input into the target video generation model, and then the target video generation model can output a video about the product after processing the image to be processed. It should be noted that the target video can be a video used to display the product.

[0109] The target video generation model can be trained in the following ways:

[0110] a1. Obtain a sample picture for a sample object and a sample background video for a background; wherein the sample picture is a picture without a background.

[0111] a2. Generate sample combined video information based on sample pictures and sample background videos;

[0112] a3. Based on the sample background video and the sample combination video, the initial video generation model is trained to obtain the target video generation model.

[0113] The information determination method provided in the embodiments of the present application can use a sample combined video generated based on a sample picture without background and a sample background video, and a target video generation model trained with a sample background video corresponding to the background, to process the picture of the target object, and can generate a video about the target object, so that the generated video ensures both the motion information of the background and the details of the target object, thereby solving the problem in the related art that the video generated for the product cannot maintain the details of the product and the background change cannot achieve coherent and regular motion, thereby improving the visual effect.

[0114] Based on the above embodiments, the present invention provides a model training device, which can be applied to Figure 1 In the model training method provided in the corresponding embodiment, refer to Figure 5 As shown, the model training device 3 may include: a first acquisition unit 31, a first processing unit 32 and a training unit 33, wherein:

[0115] A first acquisition unit 31 is used to acquire a sample picture for a sample object and a sample background video for a background; wherein the sample picture is a picture without a background;

[0116] A first processing unit 32, configured to generate a sample combined video based on the sample picture and the sample background video;

[0117] The training unit 33 is used to perform model training on the initial video generation model based on the sample background video and the sample combination video to obtain a target video generation model.

[0118] In other embodiments of the present application, the first acquisition unit 31 is further configured to perform the following steps:

[0119] Get an initial image for the sample object;

[0120] The target cutout model is used to process the initial image to obtain a sample image.

[0121] In other embodiments of the present application, the first processing unit 32 is further configured to perform the following steps:

[0122] For each frame of the sample background video, each frame is divided into multiple sub-images;

[0123] For each frame of the picture, a combined picture is generated based on any sub-picture in the plurality of sub-pictures and any picture in the sample pictures;

[0124] Generate sample composite videos based on multiple composition graphs.

[0125] In other embodiments of the present application, the training unit 33 is further configured to perform the following steps:

[0126] Perform feature extraction on each frame of the sample background video to obtain a sample feature map;

[0127] The sample feature map is processed using the first sub-initial video generation model to obtain a video to be processed;

[0128] Based on the video to be processed and the sample combination video, the second sub-initial video generation model is trained to obtain a target video generation model; wherein the initial video generation model includes the first sub-initial video generation model and the second sub-initial video generation model.

[0129] In other embodiments of the present application, the training unit 33 is further configured to perform the following steps:

[0130] Determine a target coefficient, and determine a target loss function based on the target coefficient and the basic loss function;

[0131] Based on the video to be processed, the sample combination video and the target loss function, the second sub-initial video generation model is trained to obtain the target video generation model.

[0132] In other embodiments of the present application, the training unit 33 is further configured to perform the following steps:

[0133] Determine a first pixel value of a first frame image and a second pixel value of a last frame image of a sample combined video;

[0134] A difference between the first pixel value and the second pixel value is calculated, and a target coefficient is determined based on the difference.

[0135] It should be noted that the specific description of the steps performed by each unit can be found in Figure 1 The model training method provided in the corresponding embodiment will not be repeated here.

[0136] The model training device provided in the embodiments of the present application can use a sample combination video generated based on a sample image without a background and a sample background video, and a sample background video corresponding to the background to train a target video generation model. That is, the sample background video is used when generating the target video generation model to fully ensure the motion information of the background, and the sample combination video generated based on the sample image and the sample background video is also used to ensure the diversity of data during training, thereby solving the problem that the videos generated for the products in the related technology cannot maintain the details of the products and the background changes cannot achieve coherent and regular motion, thereby improving the visual effect.

[0137] Based on the above embodiments, the present invention provides an information determination device, which can be applied to Figure 4 In the information determination method provided in the corresponding embodiment, refer to Figure 6 As shown, the information determination device 4 may include: a second acquisition unit 41 and a second processing unit 42, wherein:

[0138] The second acquisition unit 41 is used to acquire a to-be-processed picture of a target object;

[0139] A second processing unit 42 is used to process the image to be processed using a target video generation model to obtain a target video for a target object;

[0140] The target video generation model can be trained in the following way:

[0141] A sample picture for a sample object and a sample background video for a background are obtained; wherein the sample picture is a picture without a background.

[0142] Generate sample combined video information based on sample pictures and sample background videos;

[0143] Based on the sample background video and the sample combination video, the initial video generation model is trained to obtain the target video generation model.

[0144] It should be noted that the specific description of the steps performed by each unit can be found in Figure 4 The information determination method provided in the corresponding embodiment will not be repeated here.

[0145] The information determination device provided in the embodiments of the present application can use a sample combined video generated based on a sample picture without background and a sample background video, and a target video generation model trained with a sample background video corresponding to the background, to process a picture of a target object, and can generate a video about the target object, so that the generated video ensures both the motion information of the background and the details of the target object, thereby solving the problem in the related art that the video generated for the product cannot maintain the details of the product and the background change cannot achieve coherent and regular motion, thereby improving the visual effect.

[0146] Based on the above embodiments, the embodiments of the present application provide a model training device, which can be applied to Figure 1 In the model training method provided in the corresponding embodiment, refer to Figure 7 As shown, the first model training device 5 may include: a first processor 51, a first memory 52 and a first communication bus 53, wherein:

[0147] The first communication bus 53 is used to realize the communication connection between the first processor 51 and the first memory 52;

[0148] The first processor 51 is used to execute the model training program in the first memory 52 to implement the following steps:

[0149] Obtain a sample picture for a sample object and a sample background video for a background; wherein the sample picture is a picture without a background;

[0150] Generate a sample composite video based on the sample picture and the sample background video;

[0151] Based on the sample background video and the sample combination video, the initial video generation model is trained to obtain the target video generation model.

[0152] In other embodiments of the present application, the first processor 51 is used to execute the model training program in the first memory 52 to obtain a sample image for the sample object to implement the following steps:

[0153] Get an initial image for the sample object;

[0154] The target cutout model is used to process the initial image to obtain a sample image.

[0155] In other embodiments of the present application, the first processor 51 is used to execute the model training program in the first memory 52 to generate a sample combined video based on the sample picture and the sample background video to implement the following steps:

[0156] For each frame of the sample background video, each frame is divided into multiple sub-images;

[0157] For each frame of the picture, a combined picture is generated based on any sub-picture in the plurality of sub-pictures and any picture in the sample pictures;

[0158] Generate sample composite videos based on multiple composition graphs.

[0159] In other embodiments of the present application, the first processor 51 is used to execute the model training program in the first memory 52, based on the sample background video and the sample combination video, to perform model training on the initial video generation model to obtain the target video generation model, so as to implement the following steps:

[0160] Perform feature extraction on each frame of the sample background video to obtain a sample feature map;

[0161] The sample feature map is processed using the first sub-initial video generation model to obtain a video to be processed;

[0162] Based on the video to be processed and the sample combination video, the second sub-initial video generation model is trained to obtain a target video generation model; wherein the initial video generation model includes the first sub-initial video generation model and the second sub-initial video generation model.

[0163] In other embodiments of the present application, the first processor 51 is used to execute the model training program in the first memory 52, based on the video to be processed and the sample combination video, to train the second sub-initial video generation model to obtain the target video generation model, so as to implement the following steps:

[0164] Determine a target coefficient, and determine a target loss function based on the target coefficient and the basic loss function;

[0165] Based on the video to be processed, the sample combination video and the target loss function, the second sub-initial video generation model is trained to obtain the target video generation model.

[0166] In other embodiments of the present application, the first processor 51 is used to execute the determination target coefficient of the model training program in the first memory 52 to implement the following steps:

[0167] Determine a first pixel value of a first frame image and a second pixel value of a last frame image of a sample combined video;

[0168] A difference between the first pixel value and the second pixel value is calculated, and a target coefficient is determined based on the difference.

[0169] It should be noted that the specific description of the steps performed by the first processor can be referred to Figure 1 The model training method provided in the corresponding embodiment will not be repeated here.

[0170] The model training device provided in the embodiment of the present application can use the sample combination video generated according to the sample picture without background and the sample background video, and the sample background video corresponding to the background to train the target video generation model, that is, the sample background video is used when generating the target video generation model to fully ensure the motion information of the background, and the sample combination video generated according to the sample picture and the sample background video is also used to ensure the diversity of data during training, thereby solving the problem that the videos generated for the products in the related technology cannot maintain the details of the products and the background changes cannot achieve coherent and regular motion, thereby improving the visual effect.

[0171] Based on the above embodiments, the embodiments of the present application provide an information determination device, which can be applied to Figure 4 In the information determination method provided in the corresponding embodiment, refer to Figure 8 As shown, the information determination device 6 may include: a second processor 61, a second memory 62, and a second communication bus 63, wherein:

[0172] The second communication bus 63 is used to realize the communication connection between the second processor 61 and the second memory 62;

[0173] The second processor 61 is used to execute the information determination program in the second memory 62 to implement the following steps:

[0174] Get the image to be processed for the target object;

[0175] A second processing unit 42 is used to process the image to be processed using a target video generation model to obtain a target video for a target object;

[0176] The target video generation model can be trained in the following way:

[0177] A sample picture for a sample object and a sample background video for a background are obtained; wherein the sample picture is a picture without a background.

[0178] Generate sample combined video information based on sample pictures and sample background videos;

[0179] Based on the sample background video and the sample combination video, the initial video generation model is trained to obtain the target video generation model.

[0180] It should be noted that the specific description of the steps performed by the second processor can be referred to Figure 4 The information determination method provided in the corresponding embodiment will not be repeated here.

[0181] The information determination device provided in the embodiments of the present application can process the image of the target object by using a sample combined video generated based on a sample image without background and a sample background video, and a target video generation model trained with a sample background video corresponding to the background, so as to generate a video about the target object, so that the generated video ensures both the motion information of the background and the details of the target object, thereby solving the problem in the related art that the video generated for the product cannot maintain the details of the product and the background change cannot achieve coherent and regular motion, thereby improving the visual effect.

[0182] Based on the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement Figure 1 The model training method provided by the corresponding embodiment, and Figure 4 The corresponding embodiments provide steps of the information determination method.

[0183] Based on the foregoing embodiments, the embodiments of the present application provide a computer program product, including a computer program, which can be executed by the first processor 51 and the second processor 61 to complete Figure 1 The model training method provided by the corresponding embodiment, and Figure 4 The corresponding embodiments provide steps of the information determination method.

[0184] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0185] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Obtaining a sample picture for a sample object and a sample background video for a background; wherein the sample picture is a picture without a background; Generate a sample combined video based on the sample picture and the sample background video; Based on the sample background video and the sample combined video, the initial video generation model is trained to obtain a target video generation model.

2. The method according to claim 1, characterized in that The obtaining of a sample image for a sample object includes: Get an initial image for the sample object; The target cutout model is used to process the initial image to obtain the sample image.

3. The method according to claim 1, characterized in that The generating a sample combined video based on the sample picture and the sample background video includes: For each frame of the sample background video, dividing each frame of the sample background video into a plurality of sub-images; For each frame of the picture, generating a combined picture based on any sub-picture of the multiple sub-pictures and any picture of the sample pictures; The sample combined video is generated based on a plurality of component graphs.

4. The method according to claim 1, characterized in that: The step of training the initial video generation model based on the sample background video and the sample combined video to obtain a target video generation model includes: Extract features from each frame of the sample background video to obtain a sample feature map; Using the first sub-initial video generation model to process the sample feature map to obtain a video to be processed; Based on the video to be processed and the sample combined video, the second sub-initial video generation model is trained to obtain the target video generation model; wherein the initial video generation model includes the first sub-initial video generation model and the second sub-initial video generation model.

5. The method according to claim 4, characterized in that The step of training the second sub-initial video generation model based on the video to be processed and the sample combined video to obtain the target video generation model includes: Determining a target coefficient, and determining a target loss function based on the target coefficient and a basic loss function; Based on the video to be processed, the sample combination video and the target loss function, the second sub-initial video generation model is trained to obtain the target video generation model.

6. The method according to claim 5, characterized in that The determining of the target coefficient comprises: Determine a first pixel value of a first frame image and a second pixel value of a last frame image of the sample combined video; A difference between the first pixel value and the second pixel value is calculated, and the target coefficient is determined based on the difference.

7. A method for determining information, characterized in that: The method comprises: Get the image to be processed for the target object; Using a target video generation model to process the image to be processed to obtain a target video for the target object; The target video generation model may be obtained by training using the model training method described in any one of claims 1 to 6.

8. A model training device, characterized in that: The device comprises: A first acquisition unit is used to acquire a sample picture for a sample object and a sample background video for a background; wherein the sample picture is a picture without a background; A first processing unit, configured to generate a sample combined video based on the sample picture and the sample background video; The training unit is used to perform model training on the initial video generation model based on the sample background video and the sample combination video to obtain a target video generation model.

9. An information determination device, characterized in that: The device comprises: A second acquisition unit is used to acquire a to-be-processed image of a target object; A second processing unit is used to process the image to be processed using a target video generation model to obtain a target video for the target object; The target video generation model may be obtained by training using the model training method described in any one of claims 1 to 6.

10. A model training device, characterized in that: The device comprises: a first processor, a first memory and a first communication bus; The first communication bus is used to realize the communication connection between the first processor and the first memory; The first processor is used to execute the model training program in the first memory to implement the steps of the model training method as described in any one of claims 1 to 6.

11. An information determination device, characterized in that: The device comprises: a second processor, a second memory, and a second communication bus; The second communication bus is used to realize the communication connection between the second processor and the second memory; The second processor is used to execute the information determination program in the second memory to implement the steps of the information determination method as claimed in claim 7.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the model training method as described in any one of claims 1 to 6, or the steps of the information determination method as described in claim 7.

13. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 6 or claim 7.

Images