Image-to-video method and device, storage medium and program product

Abstract

The invention discloses a picture video generation method and device, a storage medium and a program product, and relates to the technical field of computers. The method comprises the following steps: acquiring an input image; performing region segmentation on the input image to obtain a main body image corresponding to a main body in the input image; generating track data corresponding to the main body image according to the main body image through a track generation model, the track data being used for indicating a motion track of a main body in the main body image in the video to be generated; and generating a motion video corresponding to the main body image according to the main body image and the trajectory data through a graph-generated video model, the motion video corresponding to the main body image being a video in which the main body in the main body image moves according to the motion trajectory corresponding to the main body image. According to the method and the device, the motion video is the video in which the subject in the subject image moves, so that the motion uniqueness of the subject in the motion video is ensured, the generation quality and the generation effect of the motion video are improved, and the stability and the controllability of the image video are also improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, storage medium and program product for generating video from an image. Background Art

[0002] The image-generated video model is a model that relies on deep learning and feature extraction technology. Users can input images and text prompt information to generate high-quality video content.

[0003] In the related art, the image and text prompt information are input into a pre-set image-generated video model, and a spatial module is used to extract the spatial features of the input image and text prompt information to generate a first feature processing result. Then, the first feature processing result is processed by the temporal self-attention module to generate a second sample feature. Finally, the motion information of the image is controlled by the motion adapter module to generate predicted video data to ensure that the processed content is more vivid and has a sense of motion.

[0004] However, in the above-mentioned image-generated video process, since the image-generated video model performs overall motion drawing on the input image, both the main area and the background area in the image will move, causing the video content to be prone to defects such as unsmoothness and unreasonableness, and the video effect is uncontrollable. Summary of the invention

[0005] The embodiments of the present application provide a method, device, storage medium and program product for generating video from an image. The technical solutions provided by the embodiments of the present application are as follows:

[0006] According to one aspect of an embodiment of the present application, a method for generating video from an image is provided, the method comprising:

[0007] Get the input image;

[0008] Performing region segmentation on the input image to obtain a subject image corresponding to a subject in the input image;

[0009] Generate trajectory data corresponding to the subject image according to the subject image through a trajectory generation model, wherein the trajectory data is used to indicate a motion trajectory of the subject in the subject image in the video to be generated;

[0010] A motion video corresponding to the subject image is generated according to the subject image and the trajectory data through an image-generated video model. The motion video corresponding to the subject image is a video in which the subject in the subject image moves along the motion trajectory corresponding to the subject image.

[0011] According to one aspect of an embodiment of the present application, a device for generating video from an image is provided, the device comprising:

[0012] An image acquisition module, used for acquiring an input image;

[0013] A region segmentation module, used to perform region segmentation on the input image to obtain a subject image corresponding to a subject in the input image;

[0014] A trajectory generation module, used to generate trajectory data corresponding to the subject image according to the subject image through a trajectory generation model, wherein the trajectory data is used to indicate a motion trajectory of the subject in the subject image in the video to be generated;

[0015] The image-generated video module is used to generate a motion video corresponding to the subject image according to the subject image and the trajectory data through an image-generated video model, wherein the motion video corresponding to the subject image is a video in which the subject in the subject image moves along the motion trajectory corresponding to the subject image.

[0016] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned image-to-video method.

[0017] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned image-to-video method.

[0018] According to one aspect of an embodiment of the present application, a computer program product is provided, wherein the computer program product comprises a computer program, and the computer program is loaded and executed by a processor to implement the above-mentioned image-to-video method.

[0019] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0020] By segmenting the subject in the input image to obtain the subject image, the trajectory data generated by the trajectory generation model is the motion data of the subject in the subject image, so that the motion video is a video of the subject in the subject image moving. Compared with the motion video generated in the related art, which is a video in which both the subject area and the background area will produce motion, the technical solution of the present application removes the background area in the input image, ensures the uniqueness of the subject motion in the motion video, thereby avoiding the unreasonable or unsmooth defects caused by the simultaneous movement of the subject and the background due to retaining the background area, thereby improving the rationality and smoothness of the motion video, that is, improving the generation quality and generation effect of the motion video, and also improving the stability and controllability of the image-generated video. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0022] Figure 2 is a flow chart of a method for generating video from an image provided by an embodiment of the present application;

[0023] Figure 3 is a flowchart of generating a motion video based on reference data provided by an embodiment of the present application;

[0024] Figure 4 is a flow chart of obtaining a subject image through defect detection and image redrawing provided by an embodiment of the present application;

[0025] Figure 5 This is an overall flow chart of a method for generating video from an image provided by an embodiment of the present application;

[0026] Figure 6 is a block diagram of an image-to-video device provided by an embodiment of the present application;

[0027] Figure 7 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0029] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment can be implemented as a picture-to-video system. The solution implementation environment may include: a terminal device 10 and a server 20.

[0030] The number of terminal devices 10 can be one or more. The terminal device 10 can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a game console, an e-book reader, a multimedia player, a wearable device, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc.

[0031] The terminal device 10 may be installed with a client of a target application, and the target application has the function of generating video from an image. When a user inputs an image in the target application, the target application generates a motion video corresponding to the input image. The present application does not limit the type of the target application, including but not limited to question-and-answer applications, music applications, video applications, game applications, social applications, etc. Optionally, the target application may be an application that needs to be downloaded and installed, or an application that can be used as a click-to-use application, and the present application does not limit this.

[0032] The server 20 is used to provide background services for the client of the target application installed and running in the terminal device 10. For example, the server 20 can be the background server of the above-mentioned target application. The server 20 can be an independent physical server, or a server cluster composed of multiple servers, or a cloud computing service center. Optionally, the server 20 provides background services for the target applications in multiple terminal devices 10 at the same time. The terminal device 10 and the server 20 can communicate with each other through the network.

[0033] In an embodiment of the present application, the image-generated video method can be implemented on the terminal device side or on the server side. The user inputs an image in the target application, and the computer device performs region segmentation on the input image to obtain a subject image corresponding to the subject in the input image, and generates trajectory data corresponding to the subject image based on the subject image through a trajectory generation model. The trajectory data is used to indicate the motion trajectory of the subject in the subject image in the video to be generated. Then, a motion video corresponding to the subject image is generated based on the subject image and the trajectory data through the image-generated video model. The motion video corresponding to the subject image is a video in which the subject in the subject image moves according to the motion trajectory corresponding to the subject image. Finally, if the quality parameter of the motion video is less than or equal to the parameter threshold, it is necessary to enhance the quality of the motion video through a video enhancement model to obtain a quality-enhanced motion video, and the terminal device returns the quality-enhanced motion video to the user in the display interface of the target application.

[0034] Please refer to Figure 2 , which shows a flow chart of a method for generating video from an image provided by an embodiment of the present application. The execution subject of each step of the method may be a computer device. The method may include at least one of the following steps 220 to 280:

[0035] Step 220, obtaining an input image.

[0036] The input image may be an image of any subject. Optionally, the subject in the input image may be a natural subject, such as a subject existing in nature such as clouds, water, trees, and flowers. Optionally, the subject in the input image may also be a human subject, and the human subject may be a human subject obtained based on an actual human image, including but not limited to human images such as ordinary people, actors, singers, athletes, entrepreneurs, and human images of film and television characters. The human subject may also be a human subject obtained based on a virtual human image, including but not limited to self-created human images such as game characters and animation characters. Optionally, the subject in the input image may also be an animal subject, and the animal subject may be an animal subject obtained based on an actual animal image, including but not limited to animal images existing in nature such as cats, dogs, lions, and fish. The animal subject may also be an animal subject obtained based on a virtual animal image, including but not limited to self-created animal images such as game characters and animation characters in animal form.

[0037] Optionally, the input image may contain text, for example, the input image may be a magazine cover image, an artist photo image, a brand poster image, etc. Optionally, the input image may not contain text, for example, the input image may be an original image without text addition processing.

[0038] Optionally, the input image may include a logo image, which is an image with logo meaning added to the original image. Generally, the logo image can be understood as a logo image. For example, if the input image is a magazine cover image, the input image may include the magazine logo image; if the input image is a brand poster image, the input image may include the brand logo image; if the input image is a campus landscape image, the input image may include the school logo image.

[0039] Step 240: perform region segmentation on the input image to obtain a subject image corresponding to the subject in the input image.

[0040] The input image includes a subject region where the subject in the input image is located and a background region where the background in the input image is located. Step 240 is used to obtain an image formed by the subject region in the input image. Exemplarily, the input image is segmented by a subject segmentation model to obtain a subject image corresponding to the subject in the input image.

[0041] The subject and background in the input image are segmented by the subject segmentation model to obtain a subject segmentation image corresponding to the subject in the input image, and the background area in the subject segmentation image is in a vacant state, that is, the subject segmentation image includes the subject area in the input image and the background area in a vacant state. The background area in the subject segmentation image in a vacant state is filled with a second color to obtain the above-mentioned subject image, that is, the subject image includes the subject area in the input image and the background area in the second color.

[0042] The present application does not limit the setting of the second color. Usually, the second color is set as a default color. For example, the second color may be white.

[0043] The subject segmentation model is a model obtained by training the image segmentation model based on training samples. Each training sample includes a sample image and a sample subject image. The sample image is subjected to regional segmentation by the image segmentation model to obtain a predicted subject image. The training goal is to reduce the difference between the predicted subject image and the sample subject image, and the parameters of the image segmentation model are adjusted to obtain the subject segmentation model. The image segmentation model here can be any publicly available image segmentation model, and this application does not limit this.

[0044] Step 260 , generating trajectory data corresponding to the subject image according to the subject image by using the trajectory generation model, wherein the trajectory data is used to indicate the motion trajectory of the subject in the subject image in the video to be generated.

[0045] According to the subject image, prompt information is generated, and the prompt information is used to prompt the trajectory generation model to generate trajectory data corresponding to the subject image. The subject image and the prompt information are input into the trajectory generation model, and the trajectory data corresponding to the subject image is output. Exemplarily, the trajectory generation model can be a multimodal large model, which is a model that combines multimodal information such as text, image, video, and audio for training. The multimodal large model here can be any publicly available multimodal large model, and this application does not limit this.

[0046] The trajectory data is used to indicate the motion trajectory of the subject in the subject image in the video to be generated. Optionally, the motion duration of the subject in the subject image indicated by the trajectory data in the video to be generated may be a default duration. For example, the default duration may be 10 seconds, and the trajectory data is used to indicate the motion trajectory of the subject in the subject image in the 10-second video to be generated. Optionally, the motion duration of the subject in the subject image indicated by the trajectory data in the video to be generated may also be determined based on the input video duration information, and the video duration information is used to indicate the duration of the video to be generated. Then, the trajectory data corresponding to the subject image is generated according to the subject image and the video duration information through the trajectory generation model, and the trajectory data is used to indicate the motion trajectory of the subject in the subject image within the duration corresponding to the video duration information.

[0047] Optionally, the trajectory data may be trajectory data obtained by the trajectory generation model predicting the motion trajectory of the subject in the subject image in the future time period after the input image, then the input image is the starting image frame of the video to be generated, and the time length of the future time period depends on the default time length or the time length corresponding to the video time length information. Optionally, the trajectory data may also be trajectory data obtained by the trajectory generation model predicting the motion trajectory of the subject in the subject image in the past time period before the input image, then the input image is the ending image frame of the video to be generated, and the time length of the past time period depends on the default time length or the time length corresponding to the video time length information. Optionally, the trajectory data may also be trajectory data obtained by the trajectory generation model predicting the motion trajectory of the subject in the subject image in the current time period where the input image is located, then the input image is the intermediate image frame in the video to be generated, and the time length of the current time period depends on the default time length or the time length corresponding to the video time length information.

[0048] In some embodiments, the trajectory generation model can obtain at least one feature point on the subject in the subject image according to the image features of the subject image, and the generated trajectory data is a trajectory sequence, and the trajectory sequence includes the motion trajectory of at least one feature point in the video to be generated. Exemplarily, if the subject in the subject image is a human subject, and the subject image is a facial image of the human subject, the trajectory generation model obtains at least one facial feature point on the human subject in the subject image according to the image features of the subject image, including but not limited to at least one mouth feature point, at least one cheek feature point, at least one eye feature point, at least one chin feature point, at least one eyebrow feature point, etc., and all the trajectory data generated by the trajectory generation model include the motion trajectory of at least one facial feature point in the video to be generated, that is, including the motion trajectory of at least one mouth feature point in the video to be generated, the motion trajectory of at least one cheek feature point in the video to be generated, the motion trajectory of at least one eye feature point in the video to be generated, the motion trajectory of at least one chin feature point in the video to be generated, the motion trajectory of at least one eye feature point in the video to be generated, etc.

[0049] Step 280, generating a motion video corresponding to the subject image according to the subject image and the trajectory data through the image-generated video model, wherein the motion video corresponding to the subject image is a video in which the subject in the subject image moves along the motion trajectory corresponding to the subject image.

[0050] According to the subject image and trajectory data, prompt information is generated, and the prompt information is used to prompt the image-generated video model to generate a motion video corresponding to the subject image. The subject image, trajectory data and prompt information are input into the image-generated video model, and the motion video corresponding to the subject image is output. The image-generated video model here can be any publicly available image-generated video model, and this application does not limit this.

[0051] Motion video is a video obtained by making the subject in the subject image move continuously according to the motion trajectory corresponding to the subject image. If the trajectory data is used to indicate the motion trajectory of the subject in the subject image in the future time period after the input image, the motion video is a video with the input image as the starting image frame, which is used to indicate the motion behavior that the subject in the subject image may take after the input image. If the trajectory data is used to indicate the motion trajectory of the subject in the subject image in the past time period before the input image, the motion video is a video with the input image as the ending image frame, which is used to indicate the motion behavior that the subject in the subject image may take before the input image. If the trajectory data is used to indicate the motion trajectory of the subject in the subject image in the current time period where the input image is located, the motion video is a video with the input image as the intermediate image frame, which is used to indicate the motion behavior that the subject in the subject image may be taking at the time node of the input image.

[0052] In some embodiments, a motion video corresponding to the video production information is generated by an image-generated video model according to the subject image, the trajectory data and the video production information, wherein the video production information includes at least one of the following: the resolution of the motion video, the frame rate of the motion video, and the duration of the motion video.

[0053] According to the subject image, trajectory data and video production information, prompt information is generated, and the prompt information is used to prompt the image-generated video model to generate a motion video corresponding to the video production information. The subject image, trajectory data, video production information and prompt information are input into the image-generated video model, and the motion video corresponding to the subject image is output.

[0054] The video production information is used to indicate the configuration information of the motion video. Among them, the resolution of the motion video refers to the number of pixels in the video screen, which is used to indicate the clarity of the motion video. The frame rate of the motion video refers to the number of screens displayed per second, which is used to indicate the smoothness of the motion video. The duration of the motion video is used to limit the time length of the motion video generated by the image-generated video model. If the motion duration of the subject in the subject image indicated by the trajectory data in the video to be generated is greater than the duration of the motion video in the video production information, then according to the duration of the motion video in the video production information, the trajectory data is cropped, and a part of the trajectory data used to generate the motion video corresponding to the video production information is obtained in the trajectory data, and then the motion video corresponding to the video production information is generated according to the subject image, a part of the trajectory data and the video production information through the image-generated video model. If the motion duration of the subject in the subject image indicated by the trajectory data in the video to be generated is less than the duration of the motion video in the video production information, when the motion video is generated according to the subject image, the trajectory data and the video production information through the image-generated video model, the motion video corresponding to the video production information is generated according to a part of the trajectory data corresponding to the last video frame in the trajectory data.

[0055] In addition, the video production information may also include information such as the saturation of the motion video, the contrast of the motion video, the brightness of the motion video, and the color temperature of the motion video.

[0056] By inputting video production information when generating a motion video, the generated motion video can meet the user's video generation needs, improve the generation quality of the motion video, and thus improve the user's satisfaction with the image-generated video.

[0057] In some embodiments, if the input data of the image-generated video model does not include video production information, a motion video corresponding to the default video production information is generated according to the default video production information. The specific value of the default video production information is set by the technician according to the video generation requirements under big data, and this application does not limit this. For example, the resolution of the motion video can be 1080p, the frame rate of the motion video can be 24FPS, and the duration of the motion video can be 10 seconds.

[0058] The technical solution provided by the embodiment of the present application obtains a subject image by segmenting the subject in the input image, so that the trajectory data generated by the trajectory generation model is the motion data of the subject in the subject image, so that the motion video is a video of the subject in the subject image moving. Compared with the motion video generated in the related art, which is a video in which both the subject area and the background area produce motion, the technical solution of the present application removes the background area in the input image, ensures the uniqueness of the subject motion in the motion video, thereby avoiding the unreasonable or unsmooth defects caused by the simultaneous movement of the subject and the background due to retaining the background area, thereby improving the rationality and smoothness of the motion video, that is, improving the generation quality and generation effect of the motion video, and also improving the stability and controllability of the image-generated video.

[0059] In some embodiments, step 260 includes at least one of sub-steps 261 - 262 .

[0060] Sub-step 261 , obtaining reference data, where the reference data is used to guide the generation of trajectory data.

[0061] In some embodiments, the reference data includes at least one of the following: reference text, reference video, and reference audio. The reference text is used to describe the motion trajectory of the subject in the subject image in the video to be generated in text form; the motion trajectory of the subject in the reference video is used to guide the generation of the motion trajectory of the subject in the subject image; and the reference audio is used to describe the motion trajectory of the subject in the subject image in the video to be generated in audio form.

[0062] Optionally, the reference text may be a text describing the movement of the subject in the subject image in a future time period after the input image. For example, if the input image is an image of a human subject without expression, the reference text may be a text describing that the subject in the subject image changes from expressionless to smiling and then to laughing in the future time period. Optionally, the reference text may also be a text describing the movement of the subject in the subject image in a past time period before the input image. For example, if the input image is an image of a human subject without expression, the reference text may be a text describing that the subject in the subject image changes from laughing to smiling and then to expressionless in the past time period. Optionally, the reference text may also be a text describing the movement of the subject in the subject image in the current time period where the input image is located. For example, if the input image is an image of a human subject without expression, the reference text may be a text describing that the subject in the subject image changes from smiling to expressionless and then to smiling in the current time period.

[0063] Optionally, the subject in the reference video and the subject in the input image may belong to the same subject category. For example, if the subject in the input image is a human subject, a reference video in which the subject is a human subject may be obtained. Optionally, the subject in the reference video and the subject in the input image may also belong to different subject categories. For example, if the subject in the input image is a human subject, a reference video in which the subject is a natural subject or an animal subject may also be obtained. The motion trajectory of the subject in the reference video is used to guide the generation of the motion trajectory of the subject in the subject image. Therefore, the motion trajectory of the subject in the subject image indicated by the trajectory data corresponding to the subject image in the video to be generated is the same as or similar to the motion trajectory of the subject in the reference video.

[0064] Based on the reference audio, a text corresponding to the reference audio is obtained, and the text corresponding to the reference audio is used to describe the motion trajectory of the subject in the subject image in the video to be generated. Optionally, the reference audio can be audio describing the motion of the subject in the subject image in a future time period after the input image, or audio describing the motion of the subject in the subject image in a past time period before the input image, or audio describing the motion of the subject in the subject image in a current time period where the input image is located.

[0065] In some embodiments, the vocal tone of the reference audio is used to guide the movement style of the subject in the subject image. For example, if the vocal tone of the reference audio is more urgent, the movement behavior of the subject in the subject image is also more urgent. If the vocal tone of the reference audio is more soothing, the movement behavior of the subject in the subject image is also slower.

[0066] In some embodiments, if the subject in the input image is a human subject, the sound production principle of the reference audio is used to guide the mouth sound production of the subject in the subject image, and the trajectory data corresponding to the subject image includes the mouth movement data of the subject in the subject image, and the mouth movement trajectory of the human subject in the generated motion video matches the sound production principle of the reference audio.

[0067] By using text, video, and audio as reference data, it is possible to enrich the generation conditions of trajectory data and increase the details of trajectory generation. Thus, the trajectory data generated by the trajectory generation model can have more possibilities and increase the diversity of image-generated videos.

[0068] Sub-step 262, generating trajectory data corresponding to the subject image according to the subject image and the reference data by using the trajectory generation model.

[0069] According to the subject image and reference data, prompt information is generated, and the prompt information is used to prompt the trajectory generation model to generate trajectory data that matches the reference data. The subject image, reference video and prompt information are input into the trajectory generation model, and the trajectory data corresponding to the subject image is output. The trajectory data corresponding to the subject image matches the motion described by the reference data.

[0070] Optionally, the reference data may include only one item of reference text, reference video, and reference audio. For example, if the reference data includes only reference text, trajectory data corresponding to the reference text is generated according to the subject image and the reference text through a trajectory generation model. The trajectory data corresponding to the reference text matches the motion trajectory of the subject in the subject image described by the reference text in the video to be generated.

[0071] Optionally, the reference data may include multiple data of reference text, reference video, and reference audio. For example, the reference text may include reference text and reference video. The trajectory data corresponding to the reference text and the reference video are generated according to the subject image, reference text and reference video through the trajectory generation model. The trajectory data corresponding to the reference text and the reference video match the motion trajectory of the subject in the subject image described by the reference text in the video to be generated, as well as the motion trajectory of the subject in the reference video.

[0072] By adding reference data when generating trajectory data corresponding to the subject image, the motion trajectory of the subject in the subject image in the video to be generated can be controlled through the reference data, so that the motion situation in the generated motion video can meet the user's expectations, enhance the trajectory details, and improve the user's satisfaction with the image-generated video.

[0073] Figure 3The flow chart of generating motion video based on reference data is shown. The subject image, reference text, reference video and reference audio are input into the trajectory generation model, and the trajectory generation model outputs the trajectory data corresponding to the subject image. The trajectory data corresponding to the subject image matches the motion trajectory described by the reference text, reference video and reference audio. The subject image, trajectory data and video production information are input into the image-generated video model, and the image-generated video model outputs the motion video corresponding to the subject image. The motion trajectory of the subject in the motion video matches the motion trajectory described by the reference text, reference video and reference audio.

[0074] In some embodiments, sub-step 262 includes at least one of sub-steps 2621 - 2622 .

[0075] Sub-step 2621, obtaining the image category of the subject image through the category detection model according to the subject image, where the image category is one of a natural subject category, a human subject category, and an animal subject category.

[0076] The category detection model is a pre-trained multi-classification model. The category detection model uses the differences in image features (such as texture, color distribution, edge characteristics, etc.) to distinguish the subject category in the subject image. The subject image is input into the category detection model, and the image category of the subject image is output. If the subject in the subject image is a natural subject, the image category of the subject image is the natural subject category. If the subject in the subject image is a human subject, the image category of the subject image is the human subject category. If the subject in the subject image is an animal subject, the image category of the subject image is the animal subject category.

[0077] For the training process of the category detection model, it is necessary to first collect multiple sample subject images with category labels, including sample natural images, sample person images and sample animal images, and then filter the sample subject images whose category labels do not conform to the corresponding image categories in the multiple sample subject images to ensure that the category labels of each filtered sample subject image are accurate. Image enhancement is performed on each filtered sample subject image, including rotation, cropping, flipping and other operations, adjusting the image size, normalizing each sample subject image, increasing data diversity, and improving the robustness of the model. After the image preprocessing is completed, a set of sample subject images with category labels is obtained. The multi-classification model includes a convolutional neural network (CNN) and a fully connected layer. The features of the sample subject image are extracted through the convolutional neural network to obtain the image features of the sample subject image, and then the sample subject image is subjected to category detection through the fully connected layer to obtain the predicted category result of the sample subject image. When the predicted category result of the sample subject image is the first result, the image category of the sample subject image is predicted to be a natural subject category, when the predicted category result of the sample subject image is the second result, the image category of the sample subject image is predicted to be a human subject category, and when the predicted category result of the sample subject image is the third result, the image category of the sample subject image is predicted to be an animal subject category. According to the difference between the predicted category result of the sample subject image and the actual category label, a multi-classification model is trained to obtain a category detection model.

[0078] The fully connected layer in the multi-classification model can also be replaced by a support vector machine (SVM), and the sample main image is classified by the support vector machine to obtain the predicted category result of the sample main image.

[0079] The present application does not limit the settings of the first result, the second result, and the third result. For example, the first result is 1, the second result is 0, and the third result is -1. When the predicted category result of the sample subject image is 1, the image category of the sample subject image is predicted to be a natural subject category. When the predicted category result of the sample subject image is 0, the image category of the sample subject image is predicted to be a human subject category. When the predicted category result of the sample subject image is -1, the image category of the sample subject image is predicted to be an animal subject category.

[0080] Exemplarily, according to the difference between the predicted category result and the actual category label of the sample subject image, the cross entropy loss function is calculated, and the Adam or SGD (Stochastic Gradient Descent) optimizer is used to adjust the weights to minimize the loss function value. Optionally, the classification effect of the category detection model can be evaluated by evaluation indicators such as accuracy, precision, recall, F1 score, etc. to improve the classification effect of the category detection model.

[0081] The subject image is input into the category detection model, and the category result of the subject image is output. The category result of the subject image is used to indicate the image category of the subject image. If the category result of the subject image is the first result, the image category of the subject image is the natural subject category. If the category result of the subject image is the second result, the image category of the subject image is the human subject category. If the category result of the subject image is the third result, the image category of the subject image is the animal subject category.

[0082] Sub-step 2622, using a trajectory generation model corresponding to the image category, based on the subject image and the reference data, generates trajectory data corresponding to the subject image, wherein the trajectory generation model corresponding to the image category is a model obtained by training the trajectory generation model based on training samples of the image category.

[0083] In the case where the image category of the subject image is a natural subject category, the trajectory data corresponding to the subject image is generated according to the subject image and the reference data through the trajectory generation model corresponding to the natural subject category, and the trajectory data corresponding to the subject image conforms to the motion characteristics of the natural subject. Among them, the trajectory generation model corresponding to the natural subject category is a model obtained by training the trajectory generation model based on the training samples of the natural subject category, and the training samples of the natural subject category include sample natural images and sample trajectory data corresponding to the sample natural images. For the training process of the trajectory generation model corresponding to the natural subject category, at least one training sample of the natural subject category is obtained, and the predicted trajectory data corresponding to the sample natural image is generated according to the sample natural image through the trajectory generation model, with the training goal of reducing the difference between the predicted trajectory data and the sample trajectory data corresponding to the sample natural image, and adjusting the parameters of the trajectory generation model to obtain the trajectory generation model corresponding to the natural subject category.

[0084] In the case where the image category of the subject image is a human subject category, the trajectory data corresponding to the subject image is generated according to the subject image and reference data through the trajectory generation model corresponding to the human subject category, and the trajectory data corresponding to the subject image conforms to the motion characteristics of the human subject. Among them, the trajectory generation model corresponding to the human subject category is a model obtained by training the trajectory generation model based on the training samples of the human subject category, and the training samples of the human subject category include sample human images and sample trajectory data corresponding to the sample human images. For the training process of the trajectory generation model corresponding to the human subject category, at least one training sample of the human subject category is obtained, and the predicted trajectory data corresponding to the sample human image is generated according to the sample human image through the trajectory generation model, with the training goal of reducing the difference between the predicted trajectory data corresponding to the sample human image and the sample trajectory data, and adjusting the parameters of the trajectory generation model to obtain the trajectory generation model corresponding to the human subject category.

[0085] In the case where the image category of the subject image is an animal subject category, the trajectory data corresponding to the subject image is generated according to the subject image and the reference data through the trajectory generation model corresponding to the animal subject category, and the trajectory data corresponding to the subject image conforms to the motion characteristics of the animal subject. Among them, the trajectory generation model corresponding to the animal subject category is a model obtained by training the trajectory generation model based on the training samples of the animal subject category, and the training samples of the animal subject category include sample animal images and sample trajectory data corresponding to the sample animal images. For the training process of the trajectory generation model corresponding to the animal subject category, at least one training sample of the animal subject category is obtained, and the predicted trajectory data corresponding to the sample animal image is generated according to the sample animal image through the trajectory generation model, with the training goal of reducing the difference between the predicted trajectory data corresponding to the sample animal image and the sample trajectory data, and adjusting the parameters of the trajectory generation model to obtain the trajectory generation model corresponding to the animal subject category.

[0086] By adopting corresponding trajectory generation models for subject images of different image categories, the generated motion trajectory can more accurately represent the motion characteristics of the subject in the subject image, so that the subject in the generated motion video moves more naturally, improving the generation quality and generation effect of the motion video.

[0087] In some embodiments, when the input data of the trajectory generation model does not include reference data, step 260 can be implemented as follows: obtaining the image category of the subject image according to the subject image through the category detection model, where the image category is one of a natural subject category, a human subject category, and an animal subject category; generating trajectory data corresponding to the subject image according to the subject image through the trajectory generation model corresponding to the image category, where the trajectory generation model corresponding to the image category is a model obtained by training the trajectory generation model based on training samples of the image category.

[0088] In some embodiments, step 280 includes sub-step 281 .

[0089] Sub-step 281, when the reference data includes reference audio, a motion video configured with reference audio is generated according to the subject image, trajectory data and reference audio through an image-generated video model; wherein, when the image category is a human subject category, the mouth movement trajectory of the human subject in the motion video matches the sound principle of the reference audio.

[0090] The subject image, trajectory data and reference audio are input into the image-generated video model, and the motion video corresponding to the subject image is output. The generated motion video corresponding to the subject image is a motion video configured with reference audio. The motion video configured with reference audio means that the audio content of the reference audio will be played synchronously during the playback of the motion video, so that the motion video is a video with sound.

[0091] In the case where the image category is a human subject category, the mouth movement trajectory of the human subject in the motion video matches the sound principle of the reference audio. Therefore, in the motion video configured with the reference audio, the reference audio appears to be the sound of the human subject in the motion video. The trajectory data input to the image-generated video model here is the trajectory data generated by the trajectory generation model based on the subject image and the reference data, and the reference data includes the reference audio.

[0092] By adding reference audio when generating motion video, the generated motion video is configured with reference audio, achieving the unification of audio and video, and matching the mouth movement trajectory of the character subject in the motion video with the sound principle of the reference audio, thereby improving the coordination between the motion video and the reference audio, thereby improving the generation quality and generation effect of the motion video.

[0093] In some embodiments, a motion video corresponding to the video production information is generated through an image-generated video model based on the subject image, trajectory data, reference audio and video production information; wherein the video production information includes at least one of the following: the resolution of the motion video, the frame rate of the motion video, and the duration of the motion video.

[0094] In some embodiments, when the reference data includes reference audio, a motion video configured with reference audio corresponding to the video production information is generated through an image-generated video model based on the subject image, trajectory data, reference audio and video production information; wherein the video production information includes at least one of the following: the resolution of the motion video, the frame rate of the motion video, and the duration of the motion video.

[0095] The subject image, trajectory data, reference audio and video production information are input into the image-generated video model, and a motion video corresponding to the subject image is output. The generated motion video corresponding to the subject image is a motion video configured with reference audio and is a motion video that conforms to the video production information. The relevant introduction of the video production information can refer to the above embodiment, which will not be repeated here.

[0096] By adding video production information on the basis of generating a motion video configured with reference audio, the generated motion video configured with reference audio can meet the user's video generation needs, improve the generation quality of the motion video, and thus improve the user's satisfaction with the image-generated video.

[0097] In some embodiments, step 280 is followed by step 290 .

[0098] Step 290: When the quality parameter of the motion video is less than or equal to the parameter threshold, a quality-enhanced motion video is generated based on the motion video by using a video enhancement model, wherein the quality parameter includes at least one of the following: resolution, frame rate, saturation, contrast, brightness, and color temperature.

[0099] Exemplarily, when the resolution of the motion video is less than or equal to the first parameter threshold, the video enhancement model generates a motion video with enhanced quality based on the motion video. When the frame rate of the motion video is less than or equal to the second parameter threshold, the video enhancement model generates a motion video with enhanced quality based on the motion video. When the saturation of the motion video is less than or equal to the third parameter threshold, the video enhancement model generates a motion video with enhanced quality based on the motion video. When the contrast of the motion video is less than or equal to the fourth parameter threshold, the video enhancement model generates a motion video with enhanced quality based on the motion video. When the brightness of the motion video is less than or equal to the fifth parameter threshold, the video enhancement model generates a motion video with enhanced quality based on the motion video. When the color temperature of the motion video is less than or equal to the sixth parameter threshold, the video enhancement model generates a motion video with enhanced quality based on the motion video.

[0100] The specific values ​​of each parameter threshold are set by the technician according to the video generation requirements, and this application does not limit this. The video enhancement model here can be any publicly available video enhancement model, and this application does not limit this.

[0101] By enhancing the quality of motion videos whose quality parameters do not meet the parameter threshold, the video quality of the motion videos is improved, so that each quality parameter of the motion videos finally outputted meets the parameter threshold, thereby ensuring the quality and effect of video generation.

[0102] In some embodiments, step 220 further includes step 230 , and step 230 includes at least one of sub-steps 231 - 234 .

[0103] Sub-step 231, performing defect detection on the input image to obtain a defect area in the input image.

[0104] The input image is subjected to defect detection by the image detection model to obtain a defect area in the input image. The defect here refers to the information added by performing the addition process in the original image of the input image, which is the information not contained in the original image of the input image. The defect area refers to the image area where the defect in the input image is located. The image detection model here can be any publicly available image detection model, and this application does not limit this.

[0105] Exemplarily, the defect includes at least one of the following: text, logo image, and the defect area includes at least one of the following: text area, logo area.

[0106] The image detection model is a pre-trained deep learning model. The image detection model is used to detect information in the input image that does not belong to the original image. The image detection model includes a text detection model and a logo image detection model. The text detection model is used to detect text in the input image, and the logo detection model is used to detect the logo image in the input image. It can also be understood that the detection model is used to detect the logo image in the input image.

[0107] In some embodiments, when the input image contains text, text detection is performed on the input image using a text detection model to obtain the text area in the input image.

[0108] The text detection model is a model obtained by training the image detection model based on text training samples. Each text training sample includes a sample text image and a sample text area in the sample text image. The sample text image is an image containing text. The image detection model is used to obtain the predicted text area in the sample text image based on the sample text image. The training goal is to reduce the difference between the predicted text area and the sample text area, and the parameters of the image detection model are adjusted to obtain the text detection model.

[0109] In some embodiments, when the input image includes a logo image, image detection is performed on the input image using a logo image detection model to obtain a logo area in the input image.

[0110] The identification image detection model is a model obtained by training the image detection model based on identification training samples. Each identification training sample includes a sample identification image and a sample identification area in the sample identification image. The sample identification image is an image containing the identification image. The image detection model is used to obtain the predicted identification area in the sample identification image based on the sample identification image. The training goal is to reduce the difference between the predicted identification area and the sample identification area, and the parameters of the image detection model are adjusted to obtain the identification image detection model.

[0111] By using corresponding image detection models to perform defect detection on the text and logo image in the input image, defect detection is made more targeted and the accuracy of defect detection is improved, thereby improving the image redrawing effect in subsequent steps.

[0112] Sub-step 232, filling the defective area in the input image with a first color to obtain a filled input image.

[0113] After detecting and obtaining a defective area in the input image, the defective area is filled with a first color to obtain a filled input image, wherein the filled input image includes other areas in the input image except the defective area and the defective area in the first color.

[0114] The present application does not limit the setting of the first color, for example, the first color may be white. The first color here and the second color used in the above step 240 may be the same color or different colors, and the present application does not limit this.

[0115] Sub-step 233, obtaining a prompt word corresponding to the input image through a visual language model according to the input image, where the prompt word corresponding to the input image is used to indicate the image content of the input image.

[0116] The input image is input into the visual language model, and the prompt word corresponding to the input image is output. The prompt word corresponding to the input image is a vocabulary used to summarize the image content of the input image.

[0117] In some embodiments, step 233 includes at least one of sub-steps 2331 - 2333 .

[0118] Sub-step 2331 , extracting features of the input image and at least one candidate prompt word respectively through a visual language model, and obtaining image features of the input image and text features corresponding to the at least one candidate prompt word respectively.

[0119] The visual language model is used to extract features of the input image to obtain image features of the input image, and to extract features of at least one candidate prompt word to obtain text features corresponding to at least one candidate prompt word. The candidate prompt word is a word in the candidate word library. The input image and at least one candidate prompt word in the candidate word library are matched to obtain the candidate prompt word with the highest matching degree, thereby obtaining the prompt word corresponding to the input image. The visual language model here can be any publicly available visual language model, and this application does not limit this.

[0120] Sub-step 2332, calculating the similarity between the image feature of the input image and the text feature corresponding to at least one candidate prompt word, and obtaining the similarity corresponding to at least one candidate prompt word.

[0121] Exemplarily, a calculation method such as cosine similarity and Euclidean distance may be used to calculate the similarity between the image feature of the input image and the text feature corresponding to at least one candidate prompt word, so as to obtain the similarity corresponding to at least one candidate prompt word.

[0122] Sub-step 2333: Determine the candidate prompt word corresponding to the maximum similarity among the similarities corresponding to at least one candidate prompt word as the prompt word corresponding to the input image.

[0123] The candidate prompt word corresponding to the maximum similarity value is the candidate prompt word with the highest matching degree with the input image among the at least one candidate prompt word. Therefore, the candidate prompt word corresponding to the maximum similarity value is determined as the prompt word corresponding to the input image.

[0124] By inferring the input image, calculating the similarity between the input image and at least one candidate prompt word, and obtaining the prompt word of the input image according to the similarity, the obtained prompt word can fully represent the image content of the input image, so that the defective area can be redrawn according to the prompt word subsequently, which can improve the redrawing quality of the defective area.

[0125] In sub-step 234, a redrawn image is generated by using an image redrawing model according to the filled input image and the prompt word corresponding to the input image. The redrawn image is an image obtained by redrawing the defective area in the filled input image.

[0126] The filled input image and the prompt word corresponding to the input image are input into the image redrawing model, and a redrawn image is output. The redrawn image is an image obtained by redrawing the defect area of ​​the filled input image in the first color based on the prompt word corresponding to the input image, and the image redrawing model only redraws the defect area in the filled input image, and does not redraw other areas except the defect area. Therefore, other areas except the defect area in the redrawn image are the same as other areas except the defect area in the input image.

[0127] In some embodiments, step 240 includes sub-step 241 .

[0128] Sub-step 241 , performing region segmentation on the redrawn image using the subject segmentation model to obtain a subject image corresponding to the subject in the redrawn image.

[0129] Figure 4 The flowchart of obtaining the main image through defect detection and image redrawing is shown. For the input image, on the one hand, text detection is performed on the input image through the text detection model to obtain the text area in the input image, and the text area in the input image is filled, and image detection is performed on the input image through the logo image detection model to obtain the logo area in the input image, and the logo area in the input image is filled to obtain the filled input image. On the other hand, the prompt word corresponding to the input image is generated through the visual language model. Then, the filled input image and the prompt word corresponding to the input image are input into the image redrawing model to generate a redrawn image, and the redrawn image is segmented to obtain the main image corresponding to the main body in the redrawn image.

[0130] By detecting defective areas in an input image and redrawing the defective areas, the main image corresponding to the main body in the redrawn image does not contain text and logo images, so that a motion video is generated based on the main image corresponding to the main body in the redrawn image, which can avoid video defects caused by text and logo images and improve the generation quality and generation effect of the motion video.

[0131] Figure 5 The overall flow chart of the image-to-video method is shown. First, the input image is defect detected. Figure 4 The flowchart shown in the figure is used to obtain the subject image. The subject image is subjected to category detection to obtain the image category of the subject image. The trajectory data corresponding to the subject image is obtained according to the subject image through the trajectory generation model corresponding to the image category. The subject image and the trajectory data are input into the image-generated video model to output the motion video corresponding to the subject image. When the quality parameter of the motion video is less than or equal to the parameter threshold, the motion video is enhanced by the video enhancement model to generate the quality-enhanced motion video. The quality-enhanced motion video is used as the final output motion video, and the image-generated video process is terminated.

[0132] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0133] Please refer to Figure 6, which shows a block diagram of an image-to-video device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned image-to-video method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the computer device described above, or it can be set in a computer device. Figure 6 As shown, the device 600 may include: an image acquisition module 610 , a region segmentation module 620 , a trajectory generation module 630 and an image-to-video module 640 .

[0134] The image acquisition module 610 is used to acquire an input image.

[0135] The region segmentation module 620 is used to perform region segmentation on the input image to obtain a subject image corresponding to the subject in the input image.

[0136] The trajectory generation module 630 is used to generate trajectory data corresponding to the subject image according to the subject image through a trajectory generation model, and the trajectory data is used to indicate the motion trajectory of the subject in the subject image in the video to be generated.

[0137] The image-generated video module 640 is used to generate a motion video corresponding to the subject image according to the subject image and the trajectory data through an image-generated video model. The motion video corresponding to the subject image is a video in which the subject in the subject image moves along the motion trajectory corresponding to the subject image.

[0138] In some embodiments, the trajectory generation module 630 is used to:

[0139] Acquiring reference data, wherein the reference data is used to guide the generation of the trajectory data;

[0140] The trajectory data corresponding to the subject image is generated according to the subject image and the reference data by using the trajectory generation model.

[0141] In some embodiments, the reference data includes at least one of the following:

[0142] A reference text, wherein the reference text is used to describe, in text form, a motion trajectory of a subject in the subject image in the video to be generated;

[0143] A reference video, wherein the motion trajectory of the subject in the reference video is used to guide the generation of the motion trajectory of the subject in the subject image;

[0144] Reference audio, where the reference audio is used to describe, in audio form, a motion trajectory of a subject in the subject image in the video to be generated.

[0145] In some embodiments, the trajectory generation module 630 is used to:

[0146] Obtaining an image category of the subject image according to the subject image through a category detection model, wherein the image category is one of a natural subject category, a human subject category, and an animal subject category;

[0147] The trajectory data corresponding to the subject image is generated according to the subject image and the reference data through the trajectory generation model corresponding to the image category, and the trajectory generation model corresponding to the image category is a model obtained by training the trajectory generation model based on the training samples of the image category.

[0148] In some embodiments, the image-to-video module 640 is used to:

[0149] In a case where the reference data includes the reference audio, generating the motion video configured with the reference audio according to the subject image, the trajectory data and the reference audio by using the image-generated video model;

[0150] Wherein, when the image category is the human subject category, the mouth movement trajectory of the human subject in the motion video matches the sound generation principle of the reference audio.

[0151] In some embodiments, the image-to-video module 640 is used to:

[0152] Generate the motion video configured with the reference audio corresponding to the video production information through the image-generated video model according to the subject image, the trajectory data, the reference audio and the video production information;

[0153] The video production information includes at least one of the following: the resolution of the motion video, the frame rate of the motion video, and the duration of the motion video.

[0154] In some embodiments, the device further comprises an image redrawing module, wherein the image redrawing module is configured to:

[0155] Performing defect detection on the input image to obtain a defect area in the input image;

[0156] Filling the defective area in the input image with a first color to obtain a filled input image;

[0157] Obtaining a prompt word corresponding to the input image according to the input image through a visual language model, wherein the prompt word corresponding to the input image is used to indicate an image content of the input image;

[0158] A redrawn image is generated according to the filled input image and the prompt word corresponding to the input image by an image redrawing model. The redrawn image is an image obtained by redrawing the defective area in the filled input image.

[0159] The region segmentation module 620 is used to perform region segmentation on the redrawn image by using a subject segmentation model to obtain a subject image corresponding to the subject in the redrawn image.

[0160] In some embodiments, the defective area includes at least one of the following: a text area and a logo area; and the image redrawing module is used to:

[0161] In the case where the input image contains text, performing text detection on the input image by using a text detection model to obtain a text area in the input image;

[0162] In the case where the input image includes a logo image, image detection is performed on the input image using a logo image detection model to obtain a logo area in the input image.

[0163] In some embodiments, the image redrawing module is used to:

[0164] Extracting features of the input image and at least one candidate prompt word respectively through the visual language model to obtain image features of the input image and text features corresponding to the at least one candidate prompt word respectively;

[0165] Calculating the similarity between the image feature of the input image and the text feature respectively corresponding to the at least one candidate prompt word, to obtain the similarity respectively corresponding to the at least one candidate prompt word;

[0166] Among the similarities respectively corresponding to the at least one candidate prompt word, the candidate prompt word corresponding to the maximum similarity is determined as the prompt word corresponding to the input image.

[0167] In some embodiments, the apparatus 600 further includes a quality enhancement module, wherein the quality enhancement module is configured to:

[0168] When the quality parameter of the motion video is less than or equal to the parameter threshold, generating a quality-enhanced motion video according to the motion video by using a video enhancement model;

[0169] The quality parameters include at least one of the following: resolution, frame rate, saturation, contrast, brightness, and color temperature.

[0170] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0171] Please refer to Figure 7 , which shows a block diagram of a computer device 700 provided in one embodiment of the present application. The computer device 700 may be any electronic device with data calculation, processing and storage functions. The computer device 700 may be used to implement the image-to-video method provided in the above embodiment.

[0172] Typically, the computer device 700 includes a processor 701 and a memory 702 .

[0173] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0174] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned image-generated video method.

[0175] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the computer device 700, and the computer device 700 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0176] In an illustrative embodiment, a computer-readable storage medium is also provided, in which a computer program is stored, and when the computer program is executed by a processor of a computer device, the above-mentioned image-to-video method is implemented. Optionally, the above-mentioned computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0177] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above-mentioned image-to-video method.

[0178] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0179] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating video from an image, characterized in that: The method comprises: Get the input image; Performing region segmentation on the input image to obtain a subject image corresponding to a subject in the input image; Generate trajectory data corresponding to the subject image according to the subject image through a trajectory generation model, wherein the trajectory data is used to indicate a motion trajectory of the subject in the subject image in the video to be generated; A motion video corresponding to the subject image is generated according to the subject image and the trajectory data through an image-generated video model. The motion video corresponding to the subject image is a video in which the subject in the subject image moves along the motion trajectory corresponding to the subject image.

2. The method according to claim 1, characterized in that The step of generating trajectory data corresponding to the subject image according to the subject image by using a trajectory generation model includes: Acquiring reference data, wherein the reference data is used to guide the generation of the trajectory data; The trajectory data corresponding to the subject image is generated according to the subject image and the reference data by using the trajectory generation model.

3. The method according to claim 2, characterized in that The reference data includes at least one of the following: A reference text, wherein the reference text is used to describe, in text form, a motion trajectory of a subject in the subject image in the video to be generated; A reference video, wherein the motion trajectory of the subject in the reference video is used to guide the generation of the motion trajectory of the subject in the subject image; Reference audio, where the reference audio is used to describe, in audio form, a motion trajectory of a subject in the subject image in the video to be generated.

4. The method according to claim 3, characterized in that The step of generating trajectory data corresponding to the subject image according to the subject image and the reference data by using the trajectory generation model includes: Obtaining an image category of the subject image according to the subject image through a category detection model, wherein the image category is one of a natural subject category, a human subject category, and an animal subject category; The trajectory data corresponding to the subject image is generated according to the subject image and the reference data through the trajectory generation model corresponding to the image category, and the trajectory generation model corresponding to the image category is a model obtained by training the trajectory generation model based on the training samples of the image category.

5. The method according to claim 4, characterized in that The step of generating a motion video corresponding to the subject image according to the subject image and the trajectory data by using the image-generated video model includes: In a case where the reference data includes the reference audio, generating the motion video configured with the reference audio according to the subject image, the trajectory data and the reference audio by using the image-generated video model; Wherein, when the image category is the human subject category, the mouth movement trajectory of the human subject in the motion video matches the sound generation principle of the reference audio.

6. The method according to claim 5, characterized in that The step of generating the motion video configured with the reference audio according to the subject image, the trajectory data and the reference audio by using the image-generated video model includes: Generate the motion video configured with the reference audio corresponding to the video production information through the image-generated video model according to the subject image, the trajectory data, the reference audio and the video production information; The video production information includes at least one of the following: the resolution of the motion video, the frame rate of the motion video, and the duration of the motion video.

7. The method according to claim 1, characterized in that After obtaining the input image, the method further includes: Performing defect detection on the input image to obtain a defect area in the input image; Filling the defective area in the input image with a first color to obtain a filled input image; Obtaining a prompt word corresponding to the input image according to the input image through a visual language model, wherein the prompt word corresponding to the input image is used to indicate an image content of the input image; generating a redrawn image according to the filled input image and the prompt word corresponding to the input image by an image redrawing model, wherein the redrawn image is an image obtained by redrawing the defective area in the filled input image; The performing region segmentation on the input image to obtain a subject image corresponding to a subject in the input image includes: The redrawn image is subjected to region segmentation using a subject segmentation model to obtain a subject image corresponding to the subject in the redrawn image.

8. The method according to claim 7, characterized in that The defective area includes at least one of the following: a text area, a logo area; The performing defect detection on the input image to obtain a defect area in the input image includes: In the case where the input image contains text, performing text detection on the input image by using a text detection model to obtain a text area in the input image; In the case where the input image includes a logo image, image detection is performed on the input image using a logo image detection model to obtain a logo area in the input image.

9. The method according to claim 7, characterized in that: The obtaining, according to the input image, a prompt word corresponding to the input image by using a visual language model includes: Extracting features of the input image and at least one candidate prompt word respectively through the visual language model to obtain image features of the input image and text features corresponding to the at least one candidate prompt word respectively; Calculating the similarity between the image feature of the input image and the text feature respectively corresponding to the at least one candidate prompt word, to obtain the similarity respectively corresponding to the at least one candidate prompt word; Among the similarities respectively corresponding to the at least one candidate prompt word, the candidate prompt word corresponding to the maximum similarity is determined as the prompt word corresponding to the input image.

10. The method according to claim 1, characterized in that After the image-generated video model is used to generate a motion video corresponding to the subject image according to the subject image and the trajectory data, the method further includes: When the quality parameter of the motion video is less than or equal to the parameter threshold, generating a quality-enhanced motion video according to the motion video by using a video enhancement model; The quality parameters include at least one of the following: resolution, frame rate, saturation, contrast, brightness, and color temperature.

11. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the image-to-video method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the image-to-video method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The computer program product comprises a computer program, which is loaded and executed by a processor to implement the image-to-video method according to any one of claims 1 to 10.

Images