Animation generation method and device and electronic equipment

Abstract

The invention provides an animation generation method and device and electronic equipment, and relates to the field of animation production. The method comprises the following steps: acquiring an original image and a target mask image; acquiring action description information, wherein the action description information is used for describing an action expected to be made by the animation character; inputting the action description information into an action generation model to obtain an action file; extracting the animation character from the original image by using the animation character area in the target mask image, and binding the joint part points of the animation character with the joint motion information of each frame to generate a target animation; the target animation is used for displaying the motion described by the motion description information made by the animation figure. The problem that the difficulty of a method for generating animations based on pictures is too high in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of animation production, and more specifically, to an animation generation method, device and electronic device. Background Art

[0002] Graffiti pictures are an important form of expression of children's paintings. Making static graffiti pictures move and turn them into animations has many practical meanings. It can be used to help children's enlightenment education, enrich their imagination, analyze their psychological growth and artistic cultivation.

[0003] At present, graffiti pictures can be converted into animation sequences through the method of manual animation production. However, the method of manual animation production is to manually draw each frame of the animation sequence based on the graffiti pictures. This is a tedious and labor-intensive task that requires art skills, a lot of time and the operation of learning complex animation software. Usually, the learning cost of many animation software is quite expensive, such as Visme, OpenToonz and Cinema 4D. Therefore, the method of manual animation production is not friendly to the general public, and the threshold and cost of animation production are obviously high.

[0004] From the above, we can see that the method of generating animation based on pictures is too difficult and has become a problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides an animation generation method, device, electronic device and storage medium, which can solve the problem that the method of generating animation based on pictures in the related art is too difficult. The technical solution is as follows:

[0006] According to one aspect of the present application, an animation generation method includes: obtaining an original image and a target mask image; the original image includes an animated character; the target mask image includes an animated character area and a background area; the animated character area is different from the background area; obtaining action description information, the action description information is used to describe the action expected to be performed by the animated character; inputting the action description information into an action generation model to obtain an action file; the action file includes at least one frame of joint motion information; the joint motion information is used to describe the movement of joint points corresponding to each action; using the animated character area in the target mask image, the animated character is extracted from the original image, and based on the joint points of the animated character, they are respectively bound to the joint motion information of each frame to generate a target animation; the target animation is used to show the animated character performing the action described by the action description information.

[0007] According to one aspect of the present application, an animation generation device includes: an image acquisition module, used to acquire an original image and a target mask image; the original image includes an animated character; the target mask image includes an animated character area and a background area; the animated character area is different from the background area; an information acquisition module, used to acquire action description information, the action description information is used to describe the action expected to be performed by the animated character; an action generation module, used to input the action description information into an action generation model to obtain an action file; the action file includes at least one frame of joint motion information; the joint motion information is used to describe the movement of joint points corresponding to each action; an animation generation module, used to extract the animated character from the original image using the animated character area in the target mask image, and generate a target animation based on the joint points of the animated character and the joint motion information of each frame; the target animation is used to show the animated character performing the action described by the action description information.

[0008] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the animation generation method as described above.

[0009] According to one aspect of the present application, a storage medium stores computer-readable instructions, and the computer-readable instructions are executed by one or more processors to implement the animation generation method as described above.

[0010] According to one aspect of the present application, a computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the animation generation method as described above.

[0011] The beneficial effects of the technical solution provided by this application are:

[0012] In the above technical scheme, the action description information is input into the action generation model to generate an action file, which can avoid manual production of action files, reduce labor costs, and reduce the difficulty of animation production; based on the joint points of the animated character in the original image and the joint motion information of each frame in the action file, the target animation can be automatically generated. The animation generation method is easy to use and does not require the user to have animator skills. Compared with manual animation, it can effectively reduce the labor cost and tool learning cost of animation production, and thus can effectively solve the problem that the method of generating animation based on pictures in related technologies is too difficult. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0014] Figure 1a is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0015] Figure 1b is a flow chart of an animation generation method according to an exemplary embodiment;

[0016] Figure 2 yes Figure 1b A flowchart of step 350 in one embodiment corresponding to the embodiment;

[0017] Figure 3 yes Figure 1b A flowchart of step 370 in one embodiment corresponding to the embodiment;

[0018] Figure 4 yes Figure 3 A flowchart of step 373 in one embodiment corresponding to the embodiment;

[0019] Figure 5 yes Figure 1b A schematic diagram of a target animation involved in the corresponding embodiment;

[0020] Figure 6 yes Figure 1b A flowchart of step 310 in one embodiment of the corresponding embodiment;

[0021] Figure 7 yes Figure 6 A flowchart of step 313 in one embodiment corresponding to the embodiment;

[0022] Figure 8 It is a schematic diagram of a specific implementation of an animation generation method in an application scenario;

[0023] Fig. 9 is a structural block diagram of an animation generating device according to an exemplary embodiment;

[0024] Fig.10 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0026] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0027] As mentioned above, graffiti images can be converted into animation sequences through manual animation production methods. However, manual animation production methods are not friendly to the general public, and the threshold and cost of animation production are obviously high.

[0028] In addition, with the rapid development of AI technology, there are also methods that automatically drive graffiti images with AI to generate animations from graffiti images. However, the AI-driven graffiti image method mainly uses a series of computer vision technologies (including target detection, character segmentation, and posture estimation) to build an automated system that converts graffiti images into animation sequences. This type of system first identifies the character and its joints from the graffiti image, and then uses the artificial bvh motion capture data to bind the character and joints to generate the animation of the graffiti character. There are two main ways to manually produce bvh motion capture data. One is to shoot the actions of real people into a video, and extract the bvh action file from the video through Rokoko or Deepmotion tools. The other is to use Blender software to create skeletal animations of performance actions, and then export them into bvh action files in Blender software. These methods require more manpower costs and time. For users, it is not easy to expand the diversity of actions. In addition, most of the character segmentation used by the system is based on classic deep learning models such as Mask R-CNN and YOLOv8. The segmentation effect of abstract graffiti images is not accurate enough, which will lead to poor binding effects of some character actions and generate unsatisfactory graffiti character animations.

[0029] As can be seen from the above, the related art still has the defect that the method of generating animation based on pictures is too difficult.

[0030] To this end, the animation generation method provided in the present application can effectively improve the accuracy of animation generation. Accordingly, the animation generation method is suitable for an animation generation device, which can be deployed in an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a laptop computer, a server, etc.

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] Figure 1a The structure of an electronic device is shown according to an exemplary embodiment. The electronic device may be a desktop computer, a notebook computer, a server, etc.

[0033] It should be noted that the electronic device is only an example adapted to the present application and cannot be considered to provide any limitation on the scope of use of the present application. The electronic device cannot be interpreted as requiring dependence on or having Figure 1a One or more components of exemplary electronic device 2000 are shown.

[0034] The hardware structure of the electronic device 2000 may vary greatly due to differences in configuration or performance, such as Figure 1a As shown, the electronic device 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0035] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .

[0036] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices.

[0037] Of course, in other examples adapted by the present application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 1a As shown, this is not intended to be a specific limitation.

[0038] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0039] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0040] The application program 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 1a (not shown), each module may respectively include computer-readable instructions for the electronic device 2000. For example, the animation generating device may be regarded as an application 253 deployed on the electronic device 2000.

[0041] The data 255 may be a photo or picture stored in a disk, or may be an original image, a target mask image, etc., stored in the memory 250 .

[0042] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the animation generation method is completed by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0043] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.

[0044] See also Figure 1b , an embodiment of the present application provides an animation generation method, which is applicable to an electronic device, which may be a desktop computer, a laptop computer, a server, or other electronic device.

[0045] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0046] like Figure 1b As shown, the method may include the following steps:

[0047] Step 310: Acquire an original image and a target mask image.

[0048] The original image includes an animated character, and the original image may be a graffiti image, for example, an image obtained by children's graffiti.

[0049] Regarding the target mask image, it is obtained by performing specific processing on the original image, for example, background cutout processing. The target mask image is used to represent a specific area, including the animation character area and the background area. Since the target mask image is a binary image, there is an extremely high contrast between the mask area (animation character area) and the non-mask area (background area) of the target mask image, that is, the animation character area is different from the background area, and this contrast makes the mask area (animation character area) very obvious.

[0050] In a possible implementation, the original image may be input into a background cutout model for processing, and pixels in the animation character area in the original image may be marked as white, and pixels in the background area may be marked as black, so as to obtain a target mask image.

[0051] Step 330, obtaining action description information.

[0052] The action description information is used to describe the action that the animated character is expected to perform.

[0053] The action description information may be obtained through user input, such as text input or language input. For example, if the user expects an animated character to perform dancing movements, then the user may input “a person dancing” to generate corresponding action description information.

[0054] Step 350: Input the action description information into the action generation model to obtain an action file.

[0055] The action file includes at least one frame of joint motion information, and the joint motion information is used to describe the motion of the joint points corresponding to each action.

[0056] Regarding action files, standard formats are usually used, such as BVH (Biovision Hierarchy), FBX (Filmbox), etc. The above formats can store complex hierarchical structures of joint points and frame-by-frame joint motion information.

[0057] First of all, it should be noted that when an animated character makes a certain action, the corresponding joint points will also move in accordance with the action.

[0058] For example, the joints of an animated character include the shoulder joint, elbow joint, and wrist joint. When the animated character waves, the shoulder joint rotates forward and upward, the arm is raised so that the arm is parallel to the ground or slightly higher, and the shoulder joint rotates left and right to drive the arm to swing; the elbow joint is slightly bent to form a certain angle between the forearm and the upper arm, and the elbow joint is properly adjusted as the shoulder joint swings to maintain the natural curvature of the arm; the wrist joint makes slight bending and rotation movements while the arm swings to increase the flexibility and naturalness of the wave. In other words, the waving action involves the coordinated movement of the shoulder joint, elbow joint, and wrist joint. The above-mentioned joint points achieve natural and coherent movements through coordinated movement. Then, by adjusting the position and angle of the above-mentioned joint points frame by frame, a realistic waving animation can be achieved.

[0059] The action generation model can be a machine learning model that has been trained and has the ability to generate action files based on action description information. In one possible implementation, the action generation model is a generative model based on the transformer architecture, the MoMask model; since the MoMask model is pre-trained with English corpus, the action description information needs to be converted into English before it is input into the MoMask model. The Chinese to English translation can be implemented based on the Opus-MT model.

[0060] In one possible implementation, such as Figure 2 As shown, step 350 may include the following steps:

[0061] Step 351, determine the action frame number of the animation character.

[0062] First, it should be noted that the action description information may also include the start time, end time, and type and details of the action. Therefore, the duration of the target animation may be preliminarily determined based on the action description information.

[0063] Furthermore, the number of action frames will affect the smoothness of the target animation that is finally generated. It is understandable that the greater the number of action frames displayed in one second, the higher the smoothness of the target animation. Common frame rates (the number of action frames displayed per second) include 24FPS: suitable for movies and animated shorts, which can provide sufficient smoothness; 30FPS: suitable for TV and most video games, providing higher smoothness; 60FPS: suitable for high-frame-rate video games and VR applications, providing the smoothest visual effects. Users can set the corresponding frame rate according to their needs. Then, based on the duration and frame rate of the animation, the total number of frames of the target animation can be calculated: Action Frames = Duration of Target Animation (seconds) × Frame Rate (FPS) Action Frames = Duration (seconds) × Frame Rate (FPS).

[0064] Step 353: input the action description information and the action frame number into the action generation model to perform text-to-action generation processing to generate joint movement information that conforms to the action frame number.

[0065] First of all, it should be noted that the action description information can be converted into frame-by-frame joint motion information through the action generation model. The joint motion information included in the action file can store the joint point position and rotation information corresponding to each frame of action. Specifically, the joint motion information can include joint point position, joint rotation angle and joint hierarchical relationship.

[0066] In a possible implementation, the process of generating an action file by the action generation model can be expressed by formula (1), where P MoMask Generate a model for the action, p Opus is the action description information, n is the number of action frames, (m1,…,m n} is p MoMask The action file generated based on the action description information and the number of action frames, m n Represents the joint motion information of the nth frame.

[0067] {m1,...,m n =P MoMask (p Opus (t prompt) ,n)...Formula (1)

[0068] For example, the process of generating a “waving” action file through an action generation model may include: creating a BVH file, setting the frame rate to 30fps, defining the joint hierarchy: shoulder joint->elbow joint->wrist joint, and writing the joint position and rotation information frame by frame. Then, the generated joint motion information may be frame 1: shoulder joint position (x1, y1, z1), rotation (rx1, ry1, rz1), frame 2: shoulder joint position (x2, y2, z2), rotation (rx2, ry2, rz2)…

[0069] Step 355, obtaining an action file based on the joint motion information of each frame.

[0070] In summary, the joint motion information of each frame generated by the action generation model can ensure that the joint motion of each frame conforms to the action described by the input action description information, so that the subsequently generated target animation can accurately display the expected animation effect.

[0071] Through the above process, there is no need to manually produce action files. The action generation model solves the problem of action file production, reduces the manpower and time costs required to increase action diversity, and can greatly reduce the manpower and learning costs of animation production. It is simple to operate and user-friendly.

[0072] Step 370 , using the animated character region of the target mask image, extracting the animated character from the original image, and binding the joint points of the animated character with the joint motion information of each frame to generate the target animation.

[0073] The target animation is used to show the animation character performing the action described by the action description information.

[0074] As mentioned above, the target mask image includes the animation character region, and the animation character region can be marked as white in the target mask image. Then, the animation character in the original image can be extracted according to the animation character region in the target mask image.

[0075] It can be understood that when the animated character performs the action described by the action description information, the movement of its joint points should conform to the joint movement information of each frame in the action description information. Therefore, before that, it is necessary to determine the joint points of the animated character. The joint points may include shoulder joints, elbow joints, wrist joints, hip joints, knee joints, ankle joints, etc., which are not limited here.

[0076] It should be noted that the original image includes all the color, texture and edge information of the animated character, while the target mask image is a binary image that only contains the outline and area of ​​the animated character and lacks detail information. Therefore, determining the joint points based on the animated character in the original image can ensure the accuracy and reliability of recognition.

[0077] Regarding the determination of joint points, in a possible implementation, before step 370, the following steps may also be included: performing target detection on the original image, determining the location area where the animated character is located in the original image, and generating a positioning area; performing posture recognition based on the animated character in the positioning area, and obtaining the joint points of the animated character in the original image.

[0078] First of all, it should be noted that a target detection method can be used to identify the position of the animated character in the original image and generate a positioning area. The positioning area can be in the form of a positioning box, which can be used to mark the position area where the animated character is located. The target detection method can be a YOLO method, a Faster R-CNN method or an SSD method, which is not limited here.

[0079] Furthermore, a posture recognition method may be used within the positioning frame to detect joint points of the animated character. The posture recognition method may be, for example, an OpenPose method, a PoseNet method, or a DensePose method, which is not limited here.

[0080] Through posture recognition, the coordinates of each joint point of the animated character can be determined, thereby determining the joint points of the animated character, so that the joint points of the animated character can be subsequently bound to the joint motion information of each frame in the action file to generate the target animation.

[0081] In one possible implementation, such as Figure 3 As shown, step 370 may include the following steps:

[0082] Step 371, mesh texturing is performed on the animated character, multiple meshes are generated on the animated character, and joint points of the animated character are associated with the meshes.

[0083] First of all, it is explained that the mesh texturing process can map the animated character onto a textured mesh, thereby generating multiple meshes on the animated character.

[0084] Furthermore, by associating the joint points of the animated character with the mesh, when the joint points move, only the corresponding mesh vertices need to be adjusted, so that natural limb movement and deformation can be achieved, thereby making the target animation more vivid, and there is no need to redraw every frame of the animated character, which can make the deformation and movement of the animated character more flexible.

[0085] In one possible implementation, mesh texturing can be achieved through the Delaunay triangulation method. Specifically, the Delaunay triangulation method is used to generate a 2D triangular mesh, which is used to perform mesh texturing on an animated character. The triangular mesh is assigned to the corresponding joint points of the animated character, thereby completing the association between the joint points of the animated character and the mesh.

[0086] Step 373, adjusting the grids associated with the joint points of the animated character according to the movement of the joint points indicated by the joint movement information, so that the animated character performs the action indicated by the action description information.

[0087] It can be understood that when an animated character makes a certain action, its joint points will also move accordingly. Furthermore, since the joint motion information describes the movement of the joint points corresponding to each action, when the joint points move, it is only necessary to adjust the corresponding mesh vertices according to the joint motion information. It should be noted that the purpose of moving the mesh vertices is to ensure that when the animated character makes a certain action, the shape and posture can be adjusted naturally with the changes in the joint points. During the movement of the joint points, complex deformations of the surface of the animated character will be caused. The adjustment of the mesh vertices can adapt to these deformations, making the action look more realistic and smooth.

[0088] For example, when an animated character is waving, the waving action mainly involves the movement of the arm, especially the movement of the shoulder joint, elbow joint and wrist joint; the joint movement information describes the movement trajectory and position change of the shoulder joint, elbow joint and wrist joint during the waving action; the corresponding mesh vertices are adjusted according to the joint movement information, including: shoulder joint movement, the movement of the shoulder joint will pull the mesh vertices associated with the shoulder joint, and the mesh vertices may slightly lift and translate forward following the movement of the shoulder joint; elbow joint movement: as the elbow joint moves outward, the mesh vertices around the elbow will also be pulled, so that the mesh shape of the upper arm part maintains a reasonable deformation while keeping consistent with the movement of the shoulder joint; wrist joint movement, the swing of the wrist joint will directly affect the mesh vertices in the hand and forearm area. These vertices need to be rotated and moved according to the movement of the wrist joint to maintain the natural shape and smoothness of the arm. By adjusting the mesh vertices corresponding to the joint points of the animated character, the mesh of the entire arm area will change with the movement of the joint points, so that the character's arm presents a natural and smooth movement effect when waving. Finally, the change of the mesh perfectly matches the movement of the joint points, making the animation look realistic and coherent.

[0089] In one possible implementation, such as Figure 4 As shown, step 370 may include the following steps:

[0090] Step 3731, determining the target position corresponding to each joint point of the animated character based on the joint motion information.

[0091] Among them, each target position is the actual position of each joint point after the animation character completes the action described by the joint movement information.

[0092] Regarding the result of the joint point movement, when the animated character completes the joint point movement indicated by each frame of joint movement information, each joint point has a corresponding end position, that is, the actual position of each joint point in the target animation of that frame. Then, the actual position of each joint point of the animated character in the target animation of that frame can be determined based on the joint movement information, thereby obtaining the target position corresponding to each joint point.

[0093] Step 3733, translate the corresponding joint points according to each target position to complete the update of the mesh associated with each joint point.

[0094] For each frame of joint motion information, the target position corresponding to the joint point in each frame can be determined, so that the associated grid is adjusted according to the target position of each joint point.

[0095] In one possible implementation, the As-Rigid-As-Possible (ARAP) method is used to update the mesh associated with each joint point. The ARAP method can simultaneously adjust the positions of other mesh vertices to keep the local shape of the mesh as unchanged as possible, thereby ensuring that the movements of the animated character are natural and realistic.

[0096] Through the above process, the joint points of the animated character can be bound to the joint motion information of each frame in the action file respectively, so as to generate a target animation frame sequence for each frame corresponding to the action described in the action description information, and then the generated target animation frame sequence can be synthesized into a continuous target animation, such as MP4 or GIF.

[0097] For example, Figure 5 A schematic diagram of a target animation is shown in FIG. Figure 5 As shown, the target animation shows GIF animation frames of an animated character dancing, and each frame is a transition of an action of the animated character dancing.

[0098] In combination with the above embodiments, the action description information is input into the action generation model to generate an action file, which can avoid manual action files, reduce labor costs, and reduce the difficulty of animation production; based on the joint points of the animated character in the original image and the joint motion information of each frame in the action file, the target animation can be automatically generated. The animation generation method is easy to use and does not require the user to have animator skills. Compared with manual animation, it can effectively reduce the labor cost and tool learning cost of animation production, and thus can effectively solve the problem of the difficulty of the method of generating animation based on pictures in related technologies. In addition, the above method is more efficient in generating target animations, and compared with other current animation production methods, it can take less time to generate the expected animation effect.

[0099] See also Figure 6 In an exemplary embodiment, step 310 may include the following steps:

[0100] Step 311, obtaining the original image.

[0101] As mentioned above, the original image includes an animated character, and the original image may be a graffiti image.

[0102] Step 313, performing background cutout processing on the original image to obtain a target mask image.

[0103] First of all, it should be noted that background cutout processing refers to separating the foreground object (animated character) from the image and separating it from the background area. Through background cutout processing, the animated character can be separated from the complex or irrelevant background, making the animated character more prominent and clear. When generating the target animation, you can focus more on the action details of the animated character without being disturbed by the background.

[0104] In addition, when generating the target animation, if the animated character has been processed with background cutout, it will be more convenient in post-editing and synthesis, and the background can be easily replaced or modified without affecting the display effect of the animated character. For example, the animated character can be synthesized into different scenes, or special effects background can be added.

[0105] Then, after background cutout processing is performed on the original image, the animation character area and the background area can be separated, the animation character area is marked as white, and the background area is marked as black, thereby obtaining a target mask image.

[0106] In one possible implementation, such as Figure 7 As shown, step 313 includes the following steps:

[0107] Step 3131, input the original image into the background cutout model to obtain a first mask image.

[0108] The first mask image includes a cartoon character region and a background region different from the cartoon character region.

[0109] In one possible implementation, the ViTMatte model or the PP-Matting model can be used to perform background matting on the original image to obtain a target mask image. It should be noted that the related technology uses semantic segmentation models such as the Mask R-CNN model and the YOLOv8 model to perform background matting, and performs classification tasks on pixels, which is a hard segmentation. The pixel values ​​of the predicted character mask are discrete (for example, the value is 0 or 1), so there will be obvious boundaries on the edges, and the edge details of the characters cannot be processed, leaving a bad visual effect. The ViTMatte model and the PP-Matting model are background matting models, and the pixels in the original image are regression tasks, which is a soft segmentation. The pixel values ​​of the predicted target mask image are continuous and smooth (for example, the value is between [0,1]), so the edge information is retained more naturally, and the edge details of the characters can be better processed.

[0110] Step 3133, using the animated character region in the first mask image to locate the animated character in the original image, and performing filling processing on the located animated character to obtain a second mask image.

[0111] The animation character region in the second mask image is different from the animation character region in the first mask image.

[0112] It can be understood that the first mask image includes the cartoon character area and a background area different from the cartoon character area. The cartoon character area is significantly different from the background area. Therefore, the cartoon character in the original image can be extracted using the cartoon character area in the first mask image.

[0113] Due to errors in the background cutout process, a small part of the pixel area in the first mask image becomes hollow and is marked as black pixels. Therefore, the animated character is filled and the area corresponding to the animated character is marked as black to obtain the second mask image. The second mask image can fill the hollow caused by the above errors and fill the edge details of the animated character, thereby ensuring that the edge pixels are continuous and smooth, thereby significantly improving the visual quality and naturalness of the subsequent target animation.

[0114] In a possible implementation, the second mask image is obtained by performing OpenCV filling processing on the animated character.

[0115] Step 3135 , selecting an image that meets the detection condition from the first mask image and the second mask image as the target mask image.

[0116] The detection condition is used to filter out a more refined target mask image, thereby reducing the impact of errors caused by background cutout processing.

[0117] In one possible implementation, step 3135 may include the following steps: respectively calculating the contour areas of the animation character region in the first mask image and the second mask image to obtain the first contour area and the second contour area; comparing the first contour area and the second contour area, and selecting the image with the larger contour area from the first mask image and the second mask image as the target mask image.

[0118] First of all, the first mask image is mask1, the corresponding first contour area is area1, the second mask image is mask2, the corresponding second contour area is area2, the detection condition is area2-area1>area2 / λ, λ is an adjustable ratio parameter, if the judgment is true, the second mask image mask2 is used as the target mask image, otherwise if the judgment is false, the first mask image mask1 is used as the target mask image. Among them, λ is set to 10 by default, which is obtained based on the estimation of the area difference between the first mask image mask1 and the second mask image mask2 when the background cutout processing fails.

[0119] Under the effect of the above-mentioned embodiment, the background cutout model can improve the effect of animation character segmentation, can better distinguish the animation character from the background area, and the generated animation character contour is more detailed. In addition, it can also ensure that the action performance of the subsequently generated target animation is more natural.

[0120] Figure 8 This is a schematic diagram of a specific implementation of an animation generation method in an application scenario. The application scenario is divided into an image processing process, an action file generation process, and an animation generation process.

[0121] like Figure 8 As shown, the image processing flow includes the following steps:

[0122] Through step 601, an original image is obtained.

[0123] Through step 603, the original image is input into the background cutout model to obtain the animated character after the background cutout processing and the first mask image mask1.

[0124] Through step 605, the animated character is filled to obtain a second mask image mask2.

[0125] Through step 607 , the contour area of ​​the animation character region in the first mask image mask1 is calculated to obtain a first contour area area1 .

[0126] Through step 609, the contour area of ​​the animation character region in the second mask image mask2 is calculated to obtain a second contour area area2.

[0127] Through step 611, a target mask image is selected from the first mask image and the second mask image using the detection condition.

[0128] like Figure 8 As shown, the action file generation process includes the following steps:

[0129] Through step 701, a text prompt word is input.

[0130] Through step 703, the text prompt word is translated into English through a machine translation model to obtain action description information.

[0131] Through step 705, the action description information is input into the action generation model to obtain an action file.

[0132] like Figure 8 As shown, the animation generation process includes the following steps:

[0133] Through step 801, the original image and the target mask image input by the image processing process and the action file input by the action file generation process are obtained.

[0134] Through step 803, the target animation is generated based on the original image, the target mask image and the action file through the Animated Drawings algorithm framework.

[0135] Among them, the Animated Drawings algorithm framework includes Delaunay triangulation and ARAP algorithm. First, Delaunay triangulation is used to generate a 2D triangular mesh, which is then used to perform mesh texturing on the animated character. The triangular mesh is assigned to the corresponding joint points of the animated character. Then, when generating the target animation, the joint points are translated and the ARAP algorithm is used to readjust the meshes associated with the joint points of the animated character, thereby binding the joint motion information of each frame of the action file and making the animated character appear to perform an action.

[0136] In this application scenario, on the one hand, the method of this application scenario is easy to use, and the operation of generating the target animation is highly automated. The user does not need to have animator skills. Compared with manual animation, it can effectively reduce the labor cost and tool learning cost of animation production; on the other hand, the method of this application scenario is more efficient in producing target animation. Compared with other current animation production methods, it can spend less time to generate the expected graffiti character animation effect. Moreover, the method of this application scenario does not require manual production of action files. The generative model of text to action solves the problem of action file production, reducing the manpower and time cost required to increase action diversity. In addition, the method of the application scenario improves the effect of animation character segmentation through the background cutout model. Compared with the current AI-driven method, it can better separate the animation character from the background area, generate a more detailed outline of the animation character, and can express a more natural animation effect.

[0137] Compared with the related art, this solution inputs the action description information into the action generation model to generate an action file, which can avoid manual action files, reduce labor costs, and reduce the difficulty of animation production; based on the joint points of the animated character in the original image and the joint motion information of each frame in the action file, the target animation can be automatically generated. This animation generation method is easy to use and does not require users to have animator skills. Compared with manual animation, it can effectively reduce the labor cost and tool learning cost of animation production, and thus can effectively solve the problem of the difficulty of the method of generating animation based on pictures in the related art. In addition, the above method is more efficient in generating target animations, and compared with other current animation production methods, it can spend less time to generate the expected animation effect.

[0138] In addition, the background cutout processing method used in this solution can improve the effect of animation character segmentation, can better distinguish the animation characters from the background area, and the generated animation character contours are more detailed. In addition, it can also ensure that the movements displayed by the subsequently generated target animation are more natural.

[0139] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0140] The following is an embodiment of the device of the present application, which can be used to execute the animation generation method involved in the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the animation generation method involved in the present application.

[0141] See also Fig. 9 In an embodiment of the present application, an animation generating device 900 is provided, including but not limited to: an image acquiring module 910 , an information acquiring module 930 , an action generating module 950 and an animation generating module 970 .

[0142] The image acquisition module 910 is used to acquire an original image and a target mask image; the original image includes an animated character; the target mask image includes an animated character region and a background region; the animated character region is different from the background region.

[0143] The information acquisition module 930 is used to acquire action description information, where the action description information is used to describe the action expected to be performed by the animated character.

[0144] The action generation module 950 is used to input the action description information into the action generation model to obtain an action file; the action file includes at least one frame of joint motion information; the joint motion information is used to describe the motion of the joint points corresponding to each action.

[0145] The animation generation module 970 is used to use the animation character area in the target mask image to extract the animation character from the original image, and bind the joint points of the animation character with the joint motion information of each frame to generate a target animation; the target animation is used to show the animation character performing the action described by the action description information.

[0146] It should be noted that the animation generating device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing animation generation. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the animation generating device will be divided into different functional modules to complete all or part of the functions described above.

[0147] In addition, the embodiments of the animation generation device and the animation generation method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0148] See also Figure 1b In an embodiment of the present application, an electronic device 4000 is provided, and the electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc.

[0149] exist Figure 1b In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0150] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 1b Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0151] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0152] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0153] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to this.

[0154] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0155] The computer-readable instructions are executed by one or more processors 4001 to implement the animation generation method in the above-mentioned embodiments.

[0156] In addition, an embodiment of the present application provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the animation generation method as described above.

[0157] A computer program product is provided in an embodiment of the present application. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the animation generation method as described above.

[0158] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An animation generation method, characterized in that: include: Acquire an original image and a target mask image; the original image includes an animated character; the target mask image includes an animated character region and a background region; the animated character region is different from the background region; Acquiring action description information, where the action description information is used to describe the action that the animated character is expected to perform; Inputting the action description information into an action generation model to obtain an action file; the action file includes at least one frame of joint motion information; The joint motion information is used to describe the motion of the joint points corresponding to each action; The animated character region in the target mask image is used to extract the animated character from the original image, and the joint points of the animated character are respectively bound to the joint motion information of each frame to generate a target animation; The target animation is used to show the animation character performing the action described by the action description information.

2. The method according to claim 1, characterized in that The obtaining of the original image and the target mask image comprises: Acquire the original image; A background cutout process is performed on the original image to obtain a target mask image.

3. The method according to claim 2, characterized in that The performing background cutout processing on the original image to obtain a target mask image includes: Inputting the original image into a background cutout model to obtain a first mask image; the first mask image includes an animation character region and a background region different from the animation character region; Using the animated character region in the first mask image to locate the animated character in the original image, and performing filling processing on the located animated character to obtain a second mask image; the animated character region in the second mask image is different from the animated character region in the first mask image; An image satisfying a detection condition is selected from the first mask image and the second mask image as the target mask image.

4. The method according to claim 3, characterized in that The selecting an image satisfying the detection condition from the first mask image and the second mask image as the target mask image includes: Calculating the contour areas of the animation character region in the first mask image and the second mask image respectively to obtain a first contour area and a second contour area; The first contour area and the second contour area are compared, and an image with a larger contour area is selected from the first mask image and the second mask image as the target mask image.

5. The method according to claim 1, characterized in that The step of inputting the action description information into an action generation model to obtain an action file comprises: Determining the number of action frames of the animated character; Inputting the action description information and the action frame number into the action generation model to perform text-to-action generation processing to generate joint motion information that conforms to the action frame number; The action file is obtained based on the joint motion information of each frame.

6. The method according to claim 1, characterized in that Before generating the target animation based on binding the joint points of the animated character with the joint motion information of each frame respectively, the method comprises: Performing target detection on the original image, determining the location area where the animation character is located in the original image, and generating a positioning area; The posture of the animated character is recognized based on the positioning area to obtain the joint points of the animated character in the original image.

7. The method according to claim 1, characterized in that The step of binding the joint points of the animated character with the joint motion information of each frame to generate a target animation comprises: Performing mesh texturing on the animated character, generating a plurality of meshes on the animated character, and associating joint points of the animated character with the meshes; The grids associated with the joint points of the animated character are adjusted according to the movement of the joint points indicated by the joint movement information, so that the animated character performs the action indicated by the action description information.

8. The method according to claim 7, characterized in that The step of adjusting the mesh associated with the joint point of the animated character according to the joint point movement indicated by the joint movement information includes: Determine the target position corresponding to each of the joint points of the animated character based on the joint motion information; each of the target positions is the actual position of each of the joint points after the animated character completes the action described by the joint motion information; The corresponding joint points are translated according to the target positions to complete the updating of the meshes associated with the joint points.

9. An animation generating device, characterized in that: include: An image acquisition module, used for acquiring an original image and a target mask image; the original image includes an animated character; the target mask image includes an animated character region and a background region; the animated character region is different from the background region; An information acquisition module, used to acquire action description information, wherein the action description information is used to describe the action expected to be performed by the animated character; An action generation module, used for inputting the action description information into an action generation model to obtain an action file; the action file includes at least one frame of joint motion information; The joint motion information is used to describe the motion of the joint points corresponding to each action; An animation generation module, used to extract the animation character from the original image by using the animation character region in the target mask image, and to bind the joint points of the animation character with the joint motion information of each frame to generate a target animation; The target animation is used to show the animation character performing the action described by the action description information.

10. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors so that the electronic device implements the animation generation method according to any one of claims 1 to 8.

Images