Image driving method and device

Abstract

The embodiment of the invention provides an image driving method, and the method comprises the steps: obtaining an initial image of a target role, and the target role comprises a target object; according to the initial image, a redrawing image is generated, and the state of at least one target object between the redrawing image and the initial image is different; and based on a preset audio, combining the initial image and the redrawing image to generate a dynamic video for the target role. According to the technical scheme, the combination mode of the initial image and the redrawing image can be controlled through the audio, and the state of the target object is accurately controlled to change along with the audio, so that the control precision, naturalness and stability of the dynamic special effect are improved, and the watching experience of a user is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of animation production technology, and in particular, to an image driving method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of information technology, image creation has become an important way for people to express their creativity and has attracted more and more attention. Image creation includes multiple steps such as drawing, coloring, animation, detail processing and post-editing. However, the current image driving technology has insufficient control accuracy for dynamic special effects, resulting in unnatural dynamic special effects and unstable effects, which affects the user's viewing experience.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0004] The embodiments of the present application provide an image driving method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0005] One aspect of an embodiment of the present application provides an image driving method, the method comprising: Acquire an initial image of a target character, wherein the target character includes a target object; generating a redrawn image according to the initial image, wherein the state of at least one target object is different between the redrawn image and the initial image; Based on the preset audio, the initial image and the redrawn image are combined to generate a dynamic video for the target character.

[0006] Optionally, the target object includes eyes and / or mouth shape, and the state of the target object includes: open, or closed.

[0007] Optionally, the image driving method further includes: generating a new redrawn image according to the initial image and the redrawn image; wherein the state of at least one target object in the new redrawn image is different from the states of the corresponding target objects in both the initial image and the redrawn image; Based on the preset audio, the initial image, the redrawn image and the new redrawn image are combined to generate a dynamic video for the target character.

[0008] Optionally, based on preset audio, the initial image and the redrawn image are combined to generate a dynamic video for the target character, including: Acquire multiple audio frames in the preset audio, where one audio frame corresponds to one audio loudness; configuring a plurality of video key frames for the plurality of audio frames according to the audio loudness of each audio frame; The multiple video key frames include the initial image and the redrawn image.

[0009] Optionally, the target object includes eyes and mouth shape, and the redrawn image includes a first redrawn image and a second redrawn image, wherein the eyes of the first redrawn image are closed and the mouth shape is closed, the eyes of the second redrawn image are open and the mouth shape is open, and the eyes of the initial image are open and the mouth shape is closed; Correspondingly, according to the audio loudness of each audio frame, multiple video key frames are configured for the multiple audio frames, including: When the audio loudness of the audio frame is less than a preset threshold, determining the initial image or the first redrawn image as a video key frame corresponding to the audio frame; When the audio loudness of the audio frame is greater than the preset threshold, the initial image or the second redrawn image is determined as a video key frame corresponding to the audio frame.

[0010] Optionally, the target object includes eyes; based on preset audio, combining the initial image and the redrawn image to generate a dynamic video for the target character, comprising: Determine a plurality of video key frames in the dynamic video according to a preset total number of video frames, wherein the plurality of video key frames include the initial image and the redrawn image; Among them, the proportion of video key frames with closed eyes in the multiple video key frames is within a preset range.

[0011] Optionally, generating a redrawn image according to the initial image includes: Based on the initial image, generating a mask image, wherein the mask image is used to define a target object to be redrawn in the initial image; The initial image and the mask image are input into a pre-trained diffusion model to obtain the redrawn image through the diffusion model.

[0012] Optionally, generating a mask image based on the initial image includes: According to the position of the target object in the initial image, a preset mask layer is marked by using a mask brush to generate the mask image; The mask image includes a marked area, and a position of the marked area in the mask image corresponds to a position of the target object in the initial image.

[0013] Another aspect of an embodiment of the present application provides an image driving device, the device comprising: An acquisition module, used for acquiring an initial image of a target character, wherein the target character includes a target object; A first generating module, configured to generate a redrawn image according to the initial image, wherein the state of at least one target object is different between the redrawn image and the initial image; The second generating module is used to combine the initial image and the redrawn image based on the preset audio to generate a dynamic video for the target character.

[0014] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0015] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0016] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0017] The above technical solution adopted in the embodiment of the present application may have the following advantages: An initial image of the target character is obtained, and a redrawn image is generated based on the initial image. The redrawn image is obtained by performing a redrawing operation on at least one target object of the target character in the initial image. According to the preset audio, the initial image and the redrawn image are combined to generate a dynamic video (such as a dynamic cartoon) for the target character. It can be seen that the embodiment of the present application controls the combination of the initial image and the redrawn image through audio, and accurately controls the state of the target object to follow the audio changes, thereby improving the control accuracy, naturalness and stability of the dynamic special effects and improving the user viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 The flowchart of the image driving method according to the first embodiment of the present application is schematically shown; Figure 2 The initial image according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows a first redrawn image according to the first embodiment of the present application; Figure 4 The second redrawing image according to the first embodiment of the present application is schematically shown; Figure 5 Schematically shows Figure 1 Flow chart of sub-steps of step S102; Figure 6 Schematically shows Figure 1 Flow chart of sub-steps of step S104; Figure 7 Schematically shows Figure 6 Flow chart of sub-steps of step S602; Figure 8 The newly added flow chart of the image driving method according to the first embodiment of the present application is schematically shown; Fig. 9 , 10 , 11, and 12 are application example diagrams of the image driving method according to the first embodiment of the present application; Fig.13 A block diagram schematically shows an image driving device according to the second embodiment of the present application; and Fig.14 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0023] First, the following terms are explained: Facial drive: The face (such as the five features) changes accordingly to the language or action. In the present application embodiment, taking a two-dimensional work (such as a comic) as an example, facial drive can refer to the comic character (such as a comic character) changing facial expressions such as opening and closing eyes, opening and closing mouth, etc. according to the plot setting.

[0024] Key frame: In computer animation, it refers to the frame where the key action of a character or object is located during the movement and change process, which is equivalent to the original painting in two-dimensional animation.

[0025] SD (Stable Diffusion) model: a diffusion model that can be used for image generation, image restoration, text generation, etc. SD Inpaint (Stable Diffusion Inpainting): an image restoration technology based on the SD model.

[0026] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: With the rapid development of information technology, image creation has become an important way for people to express their creativity and has attracted more and more attention. Image creation includes multiple steps such as drawing, coloring, animation, detail processing and post-editing.

[0027] However, the applicant has learned that the facial drive-related technologies for 2D characters still have the following defects: (1) There is a lack of two-dimensional training data. The dynamic special effects generated by driving the facial features of 2D characters may be unnatural and conflicting in style. For example, three-dimensional logic may be directly applied to the two-dimensional style to generate three-dimensional mouths, eyes, etc. for 2D characters. (2) Inaccurate recognition, especially in Q-version style or multi-person comics, the facial features of the characters cannot be accurately recognized. (3) The effect is unstable. It is impossible to drive the facial features in a more predictable manner for different angles, different characters, and different styles, and the probability of collapse is relatively high. (4) Insufficient control accuracy: It is impossible to accurately control the opening and closing of the eyes and mouth, resulting in the inability to achieve detailed expression dynamics, unpredictable effects, and multiple attempts are required to obtain satisfactory animation effects.

[0028] To this end, the embodiment of the present application provides an image driving technology solution. In this technical solution: (1) SDInpaint technology is used to accurately redraw the eyes and mouth of the character, and key frames for opening and closing the eyes and mouth of the character are generated. After that, the special effects of the character's moving mouth and eyes can be quickly generated by splicing the key frames. (2) The opening and closing of the mouth is dynamically controlled according to the loudness of the audio, so that the character's lip shape changes are synchronized with the sound output, enhancing the audio-visual consistency. (3) A 10% random eye closing probability is introduced to make the character's eye movements more natural, avoid mechanical eye opening and closing, and enhance the diversity and realism of expressions. (4) The naturalness and control accuracy of the dynamic expressions of comic characters are significantly improved, providing an efficient image driving method with stable and predictable effects. See below for details.

[0029] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0030] Embodiment 1 Figure 1 The flowchart of the image driving method according to the first embodiment of the present application is schematically shown.

[0031] like Figure 1 As shown, the image driving method may include steps S100 to S104, wherein: Step S100: acquiring an initial image of a target character, where the target character includes a target object.

[0032] Step S102: generating a redrawn image according to the initial image, wherein the state of at least one target object is different between the redrawn image and the initial image.

[0033] Step S104, based on the preset audio, combining the initial image and the redrawn image to generate a dynamic video for the target character.

[0034] The image driving method provided in this embodiment obtains the initial image of the target character and generates a redrawn image based on the initial image. The redrawn image is obtained by performing a redrawing operation on at least one target object of the target character in the initial image. According to the preset audio, the initial image and the redrawn image are combined to generate a dynamic video (such as a dynamic cartoon) for the target character. It can be seen that the embodiment of the present application controls the combination of the initial image and the redrawn image through audio, and accurately controls the state of the target object to follow the audio changes, thereby improving the control accuracy, naturalness and stability of the dynamic special effects and improving the user viewing experience.

[0035] The following combination Figure 1 , each step in steps S100~S104 and other optional steps are explained in detail.

[0036] The image driving method provided in the embodiment of the present application can be used for various types of 2D / two-dimensional / two-dimensional works, including but not limited to comics, illustrations and photos. The following will take comics as an example for a specific introduction.

[0037] Step S100 , obtaining an initial image of a target character, wherein the target character includes a target object.

[0038] The initial image can be various types of images, such as photos, paintings, screenshots, etc. Taking a comic scene as an example, the initial image can be a screenshot or original painting of the comic. The initial image can include a target character. The target character can be a person, an animal, a fantasy creature (dragon, unicorn, fairy, etc.), or other comic elements (such as a comic background). The target character can include one or more target objects. Taking a comic character as an example, the target object can be facial features (eyes, mouth, eyebrows, ears, nose), limbs (hands, feet), hairstyle, clothing, etc. The target object can have different states, and by switching between different states, the dynamic special effects of the target object can be achieved. For example, in an optional embodiment, the target object can include eyes and / or lip shape (mouth). The state of the target object can include opening or closing. When the eyes or lip shape switch between opening and closing, corresponding dynamic special effects can be generated to achieve facial drive.

[0039] Step S102 , generating a redrawn image based on the initial image, wherein the state of at least one target object is different between the redrawn image and the initial image.

[0040] The redrawn image may be obtained by performing a redraw operation on at least one target object of the target character in the initial image. The redraw operation may be used to change the state of the target object. For example, an open eye may be redrawn as a closed eye, or an open mouth may be redrawn as a closed mouth. Figures 2~4 Give an example introduction. Among them, Figure 2 Can be the initial image, Figure 3 and Figure 4 It can be a redrawn image. The target character is a person, and the target object can be the eyes and mouth of the person. Figure 2 As shown in FIG. 1 , the eyes of the person in the initial image are open and the mouth is closed. After redrawing the eyes, the first redrawn image ( Figure 3 ), the eyes and mouth of the character in the first redrawn image are closed. After redrawing the mouth, the second redrawn image can be obtained ( Figure 4 ), the eyes and mouth of the character in the second redrawn image are opened. Considering the diversity of combinations of target objects and their states, one or more different redrawn images can be generated.

[0041] The redrawing operation can be implemented by various methods, such as deep learning models, digital painting tools, manual drawing, etc. An exemplary solution is provided below.

[0042] In an optional embodiment, if Figure 5 As shown, step S102 may include: Step S500: generating a mask image based on the initial image, wherein the mask image is used to define a target object to be redrawn in the initial image.

[0043] Step S502: input the initial image and the mask image into a pre-trained diffusion model to obtain the redrawn image through the diffusion model.

[0044] Exemplarily, the target object in the initial image can be identified and the area where it is located can be determined by a variety of methods. For example: manual selection, edge detection, region segmentation algorithm or deep learning model. Based on the area where the target object is located, a mask image can be generated. The mask image can be a binary image of the same size as the initial image. The mask image can be divided into an area to be redrawn and an area not to be redrawn, and the two can be distinguished by different colors (values). The area to be redrawn corresponds to the area where the target object in the initial image is located, and is used to identify the target object to be redrawn.

[0045] By inputting the initial image and the mask image into a pre-trained diffusion model (SD model), the SDInpaint technology can be used to accurately redraw the target object to be redrawn, and a redrawn image that is natural and stable and matches the style of the original (initial image) can be obtained.

[0046] In this embodiment, the mask image can enable the diffusion model to quickly determine the target object to be redrawn in the initial image and generate a redrawn image consistent with the style of the initial image, which can effectively improve the efficiency, accuracy and stability of redrawing, reduce the probability of collapse, and adapt to redrawing at different angles, different characters, and different styles.

[0047] In an optional embodiment, step S500 may include: annotating a preset mask layer using a mask brush according to the position of the target object in the initial image to generate the mask image, wherein the mask image includes an annotated area, and the position of the annotated area in the mask image corresponds to the position of the target object in the initial image.

[0048] Exemplarily, a mask layer with the same size as the initial image can be created in advance. The position (region) of the target object in the initial image is identified and determined, and a mask image can be generated by marking on the preset mask layer using a mask brush. The mask image can include a marked region, which is the region to be redrawn and can be used to identify the target object in the initial image.

[0049] In this embodiment, the mask brush can be used to freely select the area to be redrawn, providing a flexible annotation method, accurately controlling the target object to be redrawn, and further improving the precision and accuracy of redrawing. The mask image can also effectively isolate the redrawn area and the non-redrawn area to avoid destroying the comic elements of non-target objects, thereby optimizing the visual effect of the redrawn image.

[0050] The above embodiments exemplarily introduce how to change the state of a target object. The following will further introduce how to generate dynamic effects by combining target objects in different states to achieve the driving of a target character.

[0051] Step S104 , based on preset audio, the initial image and the redrawn image are combined to generate a dynamic video for the target character.

[0052] The preset audio may be one or more of dubbing, background music, narration, ambient sound effects, and special sound effects. Based on the preset audio, the initial image and the redrawn image may be spliced ​​as video key frames to obtain a dynamic video for the target character. For example, each audio frame of the preset audio may be subjected to emotion analysis (such as happiness, sadness, anger, etc.), and the combination method may be determined according to the emotion change. For example, when the emotion corresponding to an audio frame is happiness, the character is more inclined to open his eyes and mouth, so the second redrawn image ( Figure 4 ) as the video key frame corresponding to the audio frame. When the emotion corresponding to an audio frame is sad, the character is more likely to close his eyes and mouth, so the first redrawn image ( Figure 3 ) as the probability of the video key frame corresponding to the audio frame increases. Of course, redrawn images or initial images in other states can also be selected as corresponding video key frames to increase randomness and avoid mechanization, so as to make the character's expression more natural and delicate. In some embodiments, based on the type of emotion, the selection and combination of video key frames can be further optimized according to the intensity of emotion. In addition to emotional factors, the selection and combination of video key frames can also be determined according to the audio content, audio rhythm beat, and audio loudness of each audio frame to accurately control the state change of the target object, generate more natural dynamic special effects, and ensure audio-visual synchronization and audio-visual effects. An exemplary scheme will be provided below.

[0053] In an optional embodiment, if Figure 6 As shown, step S104 may include: Step S600: Acquire multiple audio frames in the preset audio, where one audio frame corresponds to one audio loudness.

[0054] Step S602: configuring a plurality of video key frames for the plurality of audio frames according to the audio loudness of each audio frame, wherein the plurality of video key frames include the initial image and the redrawn image.

[0055] Exemplarily, multiple audio frames in the preset audio can be obtained, and the audio loudness of each audio frame can be determined. Audio loudness refers to the volume, which can be quantified by sound pressure or decibels. According to the audio loudness of each audio frame, video key frames can be configured for multiple audio frames one by one, and finally a dynamic video for the target character, i.e., a dynamic comic, can be generated. If the state of a target object changes in different video key frames played continuously, a dynamic special effect of the target object can be formed, thereby visually realizing the driving of the target object (such as facial features driving, limb driving, etc.).

[0056] In this embodiment, the target object is controlled and driven by audio loudness so that the state change of the target object matches the sound output, which can enhance the audio-visual consistency, improve the naturalness of the facial expressions of comic characters, and effectively optimize the user viewing experience.

[0057] The specific configuration rules of the video key frames can be set according to actual needs, and an exemplary solution will be provided below.

[0058] In an optional embodiment, the target object may include eyes and mouth shape, and the redrawn image may include a first redrawn image and a second redrawn image. The first redrawn image may have eyes in a closed state and mouth shape in a closed state, such as Figure 3 The eye state of the second redrawn image can be open, and the lip state can be open, such as Figure 4The eye state of the initial image may be open, and the mouth state may be closed, as shown in FIG. Figure 2 Correspondingly, Figure 7 As shown, step S602 may include: Step S700: when the audio loudness of the audio frame is less than a preset threshold, determine the initial image or the first redrawn image as a video key frame corresponding to the audio frame.

[0059] Step S702: when the audio loudness of the audio frame is greater than the preset threshold, determine the initial image or the second redrawn image as a video key frame corresponding to the audio frame.

[0060] Exemplarily, the audio loudness of the audio frame can be compared with a preset threshold. When it is greater than the preset threshold, that is, when the volume is high, the mouth shape and eye shape of the character are more likely to be open, so the initial image or the first redrawn image can be determined as the video key frame corresponding to the audio frame, or the probability of selecting the two can be increased. Correspondingly, when it is less than the preset threshold, that is, when the volume is low, the mouth shape of the character is more likely to be closed, so the initial image or the first redrawn image can be determined as the video key frame corresponding to the audio frame, or the probability of selecting the two can be increased.

[0061] In this embodiment, the opening and closing of the eyes and mouth of the character is controlled according to the audio loudness, so that the eye and mouth movements of the character can change synchronously with the sound output, ensuring audio-visual synchronization. This algorithm is completed automatically without manual intervention, which can further improve the production efficiency of dynamic comics.

[0062] The above embodiments describe how to combine the initial image and the redrawn image based on the preset audio to create a dynamic video for the target character. In practical applications, randomness, user input, narrative logic, etc. can also be considered to optimize the selection and combination method of the video key frames to create a richer, more interactive, and more personalized dynamic video, and further optimize the user experience. An exemplary solution is provided below.

[0063] In an optional embodiment, the image driving method may further include: determining a plurality of video key frames in the dynamic video according to a preset total number of video frames, the plurality of video key frames including the initial image and the redrawn image; wherein a proportion of video key frames with closed eyes in the plurality of video key frames is within a preset range.

[0064] Exemplarily, a random eye-closing mechanism can be introduced. The initial image and the redrawn image are randomly arranged, wherein the proportion of video key frames with closed eyes in multiple video key frames corresponding to the dynamic video is set within a preset range, for example, 10% of the total number of frames of the dynamic video. That is, introducing a 10% random eye-closing probability can make the eye movements of the characters more natural, avoid mechanical continuous eye opening and closing, and increase the diversity and realism of expressions. In some embodiments, the random eye-closing mechanism can also be introduced based on the consideration of preset audio to optimize the configuration rules of video key frames.

[0065] In this embodiment, the eye driving effect can be optimized through the random eye closure mechanism, and the expressiveness and naturalness of dynamic special effects can be improved.

[0066] The above embodiments describe how to achieve dynamic special effects by splicing the initial image and redrawing the image. In practical applications, in order to save resources, the redrawn image can be only for the common state of the target object, and the uncommon state can be not redrawn, so as to achieve dynamic effects with less redrawn images. However, this solution still has room for further optimization, and an exemplary solution will be provided below.

[0067] In an optional embodiment, if Figure 8 As shown, the image driving method may further include: Step S800, generating a new redrawn image according to the initial image and the redrawn image; the state of at least one target object in the new redrawn image is different from the states of the corresponding target objects in the initial image and the redrawn image.

[0068] Step S802: Based on the preset audio, the initial image, the redrawn image and the new redrawn image are combined to generate a dynamic video for the target character.

[0069] For example, the character's eyes have two states: open and closed, and the mouth shape has two states: open and closed, for a total of four combinations. Among them, the closed eyes and open mouth shape is a less common state and can be omitted. Based on the initial image (open eyes and closed mouth), the first redrawn image (closed eyes and closed mouth) and the second redrawn image (open eyes and closed mouth) are generated, such as Figures 2~4 As shown. The three images are stitched together as video keyframes to generate dynamic effects for the eyes and mouth. In order to further optimize the dynamic effect, a new redrawn image (uncommon state combination, such as closed mouth and closed eyes) can be generated based on the initial image, the first redrawn image, and the second redrawn image. Then, according to the preset audio, the four images are combined to further enrich the dynamic expressions of the characters and enhance the user's viewing experience. There are many ways to generate new redrawn images, including but not limited to deep learning models, digital painting tools, and manual drawing.

[0070] In this embodiment, more redrawn images are generated and added to the splicing, which can further optimize the dynamic special effects and enhance the user's viewing experience.

[0071] In order to make this application easier to understand, the following Figures 9 to 12 An exemplary application is provided.

[0072] S1: Acquire an initial image of a target character, where the target character may include one or more target objects.

[0073] The target character can be a person, the target object can be the eyes and mouth shape of the person, and the initial image can be Figure 2 shown.

[0074] S2: Based on the initial image, the target object is marked using the mask brush to obtain a mask image.

[0075] S3: Input the initial image and the mask image into the pre-trained SD model, and input the first redrawn image and the second redrawn image through the SD model, such as Figure 3 and Figure 4 shown.

[0076] S4: Acquire multiple audio frames of the preset audio, and determine the loudness of each audio frame.

[0077] S5: According to the loudness of each audio frame, multiple video key frames are configured for multiple audio frames to generate a dynamic video for the target character. The specific configuration process may also introduce a random eye closure mechanism, for example, the video key frames with closed eyes account for 10% of the total number of frames of the dynamic video.

[0078] In this exemplary application: (1) SD Inpaint technology is used to accurately redraw the eyes and mouth of the character, and key frames for opening and closing the eyes and mouth of the character are generated. After that, special effects of moving the mouth and eyes of the character can be quickly generated by splicing key frames. (2) The opening and closing of the mouth is dynamically controlled according to the loudness of the audio, so that the changes in the character's mouth shape are synchronized with the sound output, enhancing the audio-visual consistency. (3) A 10% random eye closing probability is introduced to make the character's eye movements more natural, avoid mechanical eye opening and closing, and enhance the diversity and realism of expressions. (4) The naturalness and control accuracy of the dynamic expressions of comic characters are significantly improved, realizing an efficient image-driven method with stable and predictable effects.

[0079] Embodiment 2 Fig.13The block diagram of the image driving device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.13 As shown, the apparatus 1000 may include: an acquisition module 1100, a first generation module 1200 and a second generation module 1300, wherein: An acquisition module 1100 is used to acquire an initial image of a target character, where the target character includes one or more target objects; A first generating module 1200 is used to generate a redrawn image according to the initial image, wherein the state of at least one target object is different between the redrawn image and the initial image; The second generating module 1300 is used to combine the initial image and the redrawn image based on the preset audio to generate a dynamic video for the target character.

[0080] As an optional embodiment, the target object includes eyes and / or mouth shape, and the state of the target object includes: open, or closed.

[0081] As an optional embodiment, the image driving device 1000 is further used for: generating a new redrawn image according to the initial image and the redrawn image; wherein the state of at least one target object in the new redrawn image is different from the states of the corresponding target objects in both the initial image and the redrawn image; Based on the preset audio, the initial image, the redrawn image and the new redrawn image are combined to generate a dynamic video for the target character.

[0082] As an optional embodiment, based on the preset audio, the initial image and the redrawn image are combined to generate a dynamic video for the target character, including: Acquire multiple audio frames in the preset audio, where one audio frame corresponds to one audio loudness; configuring a plurality of video key frames for the plurality of audio frames according to the audio loudness of each audio frame; The multiple video key frames include the initial image and the redrawn image.

[0083] As an optional embodiment, the target object includes eyes and mouth shape, and the redrawn image includes a first redrawn image and a second redrawn image, wherein the eyes of the first redrawn image are closed and the mouth shape is closed, the eyes of the second redrawn image are open and the mouth shape is open, and the eyes of the initial image are open and the mouth shape is closed; Correspondingly, according to the audio loudness of each audio frame, multiple video key frames are configured for the multiple audio frames, including: When the audio loudness of the audio frame is less than a preset threshold, determining the initial image or the first redrawn image as a video key frame corresponding to the audio frame; When the audio loudness of the audio frame is greater than the preset threshold, the initial image or the second redrawn image is determined as a video key frame corresponding to the audio frame.

[0084] As an optional embodiment, the target object includes eyes; based on preset audio, combining the initial image and the redrawn image to generate a dynamic video for the target character includes: Determine a plurality of video key frames in the dynamic video according to a preset total number of video frames, wherein the plurality of video key frames include the initial image and the redrawn image; Among them, the proportion of video key frames with closed eyes in the multiple video key frames is within a preset range.

[0085] As an optional embodiment, generating a redrawn image according to the initial image includes: Based on the initial image, generating a mask image, wherein the mask image is used to define a target object to be redrawn in the initial image; The initial image and the mask image are input into a pre-trained diffusion model to obtain the redrawn image through the diffusion model.

[0086] As an optional embodiment, generating a mask image based on the initial image includes: According to the position of the target object in the initial image, a preset mask layer is marked by using a mask brush to generate the mask image; The mask image includes a marked area, and a position of the marked area in the mask image corresponds to a position of the target object in the initial image.

[0087] Embodiment 3 Fig.14The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the image driving method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.14 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of an image driving method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0088] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0089] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0090] It should be pointed out that Fig.14 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0091] In this embodiment, the image driving method stored in the memory 10010 may also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0092] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the image driving method in the embodiment are implemented.

[0093] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the image driving method in the embodiment, etc. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0094] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0095] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0096] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image driving method, characterized in that: The method comprises: Acquire an initial image of a target character, wherein the target character includes a target object; generating a redrawn image according to the initial image, wherein the state of at least one target object is different between the redrawn image and the initial image; Based on the preset audio, the initial image and the redrawn image are combined to generate a dynamic video for the target character.

2. The method according to claim 1, characterized in that The target object includes eyes and / or mouth shape, and the state of the target object includes: open or closed.

3. The method according to claim 1, characterized in that Also includes: generating a new redrawn image according to the initial image and the redrawn image; wherein the state of at least one target object in the new redrawn image is different from the states of the corresponding target objects in both the initial image and the redrawn image; Based on the preset audio, the initial image, the redrawn image and the new redrawn image are combined to generate a dynamic video for the target character.

4. The method according to claim 1, characterized in that: Based on the preset audio, the initial image and the redrawn image are combined to generate a dynamic video for the target character, including: Acquire multiple audio frames in the preset audio, where one audio frame corresponds to one audio loudness; configuring a plurality of video key frames for the plurality of audio frames according to the audio loudness of each audio frame; The multiple video key frames include the initial image and the redrawn image.

5. The method according to claim 4, characterized in that The target object includes eyes and mouth shape, and the redrawn image includes a first redrawn image and a second redrawn image, wherein the eyes of the first redrawn image are closed and the mouth shape is closed, the eyes of the second redrawn image are open and the mouth shape is open, and the eyes of the initial image are open and the mouth shape is closed; Correspondingly, according to the audio loudness of each audio frame, multiple video key frames are configured for the multiple audio frames, including: When the audio loudness of the audio frame is less than a preset threshold, determining the initial image or the first redrawn image as a video key frame corresponding to the audio frame; When the audio loudness of the audio frame is greater than the preset threshold, the initial image or the second redrawn image is determined as a video key frame corresponding to the audio frame.

6. The method according to claim 1, characterized in that The target object includes eyes; based on the preset audio, the initial image and the redrawn image are combined to generate a dynamic video for the target character, including: Determine a plurality of video key frames in the dynamic video according to a preset total number of video frames, wherein the plurality of video key frames include the initial image and the redrawn image; Among them, the proportion of video key frames with closed eyes in the multiple video key frames is within a preset range.

7. The method according to any one of claims 1 to 6, characterized in that: Generating a redrawn image according to the initial image includes: Based on the initial image, generating a mask image, wherein the mask image is used to define a target object to be redrawn in the initial image; The initial image and the mask image are input into a pre-trained diffusion model to obtain the redrawn image through the diffusion model.

8. The method according to claim 7, characterized in that Based on the initial image, a mask image is generated, comprising: According to the position of the target object in the initial image, a preset mask layer is marked by using a mask brush to generate the mask image; The mask image includes a marked area, and a position of the marked area in the mask image corresponds to a position of the target object in the initial image.

9. An image driving device, characterized in that: The device comprises: An acquisition module, used for acquiring an initial image of a target character, wherein the target character includes a target object; A first generating module, configured to generate a redrawn image according to the initial image, wherein the state of at least one target object is different between the redrawn image and the initial image; The second generating module is used to combine the initial image and the redrawn image based on the preset audio to generate a dynamic video for the target character.

10. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 8 are implemented.

Images