Software-defined method and system for real-time automatic generation of children's animation

By building an animation knowledge base and natural language understanding, combined with animation planning, real-time generation of children's animations has been achieved, solving the problems of existing systems being unable to generate animations quickly and in terms of ease of use, and meeting the animation creation needs of children.

CN116012496BActive Publication Date: 2026-11-17EAST CHINA NORMAL UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202211621355.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-16
Publication Date
2026-11-17
Estimated Expiration
2042-12-16

AI Technical Summary

Technical Problem

Existing children's animation generation systems cannot achieve rapid, real-time generation from voice to animation, and have a high barrier to entry for child users, making it impossible to achieve completely uninterrupted and flexible adjustments.

Method used

By building an animation knowledge base and combining natural language understanding and animation planning, animations are generated in real time, including speech recognition and error correction, spatiotemporal planning of animation objects, and automatic assembly of materials, using the animation database in Unity to generate animations.

Benefits of technology

It achieves real-time generation and ease of use of children's animation, meets the real-time and ease-of-use requirements of the children's animation generation process, and lowers the threshold for animation creation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116012496B_ABST
    Figure CN116012496B_ABST
Patent Text Reader

Abstract

This invention discloses a software-defined method and system for real-time automatic generation of children's animation. The animation generation method includes: constructing an animation knowledge base; correcting errors in the speech recognition results input by children and extracting animation scripts from children's natural language text; performing spatiotemporal planning of animation objects based on the animation scripts and the animation knowledge base to obtain an animation planning spatiotemporal knowledge graph; obtaining animation specifications for final animation display based on the animation planning spatiotemporal knowledge graph; and assembling the animation according to the animation specifications. The animation generation system includes an interaction layer, a functional layer, a knowledge layer, and a data persistence layer. The interaction layer handles interaction with children, the functional layer is responsible for real-time automatic animation generation, the knowledge layer stores the knowledge required for animation planning, and the data persistence layer stores intermediate result files of animation planning. This method and system achieve fully automatic conversion from stories described by children's natural language to animated films.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer-aided animation automatic generation, specifically involving a software-defined method and system for real-time automatic generation of children's animation, which includes natural language understanding, script generation, animation planning, and animation knowledge base. Background Technology

[0002] Animation, as a vivid medium, has always been loved by the public, especially children. Transforming children's stories into animations and having them create them can significantly enhance their creativity and language skills. Furthermore, according to Feynman's output learning method, good output can better drive input learning; children creating animations as an form of output will undoubtedly bring new experiences to children's education. However, animation creation has always been a high-barrier-to-entry process. Typically, producing an animation requires a significant investment of time and effort from professionals such as character designers, scene designers, animators, and animation directors. Modern animation production often requires the use of specialized software such as Maya and Blender. This presents a significant challenge for children.

[0003] Automatic animation generation systems for children face challenges such as the complexity of natural language and the real-time nature of planning. They need to address issues like correctly understanding user intent, arranging animation content logically and aesthetically, generating animations quickly, and ensuring system ease of use. In recent years, with the continuous development of information technology, many software programs and systems have emerged to assist in the generation of animations, such as semi-automatic animation generation software like Flash, HTML5, and Adobe Animate, the fully automatic animation generation system "Swan," and the intelligent animation creation system PNAI. However, these systems typically cannot generate animations without any human intervention, and the input methods are subject to many limitations. After generating initial content, a certain amount of time is required to modify and adjust the animation content, making it impossible to achieve rapid, real-time generation from speech to animation. These semi-automatic or fully automatic human-assisted animation generation systems still present a high barrier to entry for children. Summary of the Invention

[0004] To address the challenges of automatically generating children's animations, this invention aims to provide a method and system for real-time animation generation based on children's voice. The approach involves obtaining script segments primarily focused on character actions through natural language understanding; combining this with an animation knowledge base to perform spatiotemporal planning on the script, defining the spatiotemporal states of animation objects, and thus obtaining an animation planning spatiotemporal knowledge graph; generating an animation specification file based on the knowledge graph; and finally, retrieving material information from the animation database in Unity based on the specification file information, automatically assembling the animation, and presenting it to the user.

[0005] The specific technical solution for achieving the objective of this invention is as follows:

[0006] A software-defined method for real-time automatic generation of children's animations includes the following steps:

[0007] Step 1: Build an animation knowledge base;

[0008] Step 2: Correct the speech recognition results input by the child and extract the animation script from the child's natural language text;

[0009] Step 3: Based on the animation script and combined with the animation knowledge base, perform spatiotemporal planning of the animation objects to obtain the animation planning spatiotemporal knowledge graph;

[0010] Step 4: Obtain the animation specifications that can be used for the final animation display based on the animation planning spatiotemporal knowledge graph;

[0011] Step 5: Assemble the animation according to the animation specifications.

[0012] Step one specifically involves constructing the necessary knowledge base for animation planning, including a material knowledge base, a layout knowledge base, and a motion knowledge base. The material knowledge base includes the mapping relationship between materials and names (material name mapping table), material information such as category, size, and orientation (material information table), and specific materials (material library); the layout knowledge base includes a layout area rule base and a point-taking strategy rule base; the motion knowledge base includes the mapping relationship between actions and categories (action category mapping table), the mapping relationship between compound actions and atomic actions (compound action splitting table), and action trajectory splitting rules.

[0013] Step two specifically involves: correcting the speech recognition results input by children, which means correcting the speech recognition text in three aspects: word segmentation, part of speech, and word content, based on the content of the material library; extracting the animation script from the children's natural language text, which means splitting the corrected text into short sentences based on predicate verbs, and then extracting background, character, object, action sequence, and location marker information from each short sentence based on basic sentence patterns to generate the corresponding animation script segment.

[0014] The third step is as follows: Based on the animation script obtained in the second step, and combined with the animation knowledge base, spatiotemporal planning is performed on the animation objects to obtain the action execution time and execution trajectory of each animation object in the current script, and this information is used to generate an animation planning spatiotemporal knowledge graph.

[0015] Step four specifically involves generating an animation specification file corresponding to the currently input sentence based on the quantitative information in the animation planning spatiotemporal knowledge graph from step three, combined with other information in the scene. After the user inputs all sentences, the specification files corresponding to each sentence are merged to obtain the specification file for displaying the overall animation.

[0016] Step five specifically involves retrieving background, animation object appearance, and motion animation file materials from the animation material library based on the material information and spatiotemporal information in the animation specifications, and assembling and displaying them in Unity.

[0017] A system for implementing the above method, the system comprising: an interaction layer, a functional layer, a knowledge layer, and a data persistence layer.

[0018] The interaction layer handles the interaction between children and the system, including a voice interaction module, a touch interaction module, a resource display module, and an animation display module. The voice interaction module is responsible for acquiring the child's voice information; the touch interaction module is responsible for acquiring the child's touch operations such as clicks and swipes; the resource display module is responsible for displaying the resources in the resource library to the child in the form of thumbnails; and the animation display module will play and display the generated animations to the child.

[0019] The functional layer handles the automatic generation of animations, including a script extraction module, an animation planning module, and an animation assembly module. The script extraction module is responsible for speech recognition and error correction, extraction of entities and entity relationships, and extraction of action sequences; the animation planning module is responsible for planning the time, actions, arrangement, and paths of each animation object in the animation, and generating animation specifications; the animation assembly module is responsible for finding and calling animation materials, generating subtitles and voice-over, and obtaining complete animation content.

[0020] The knowledge layer stores the knowledge base needed for animation generation, including the material knowledge base, the layout knowledge base, and the motion knowledge base.

[0021] The data persistence layer stores various intermediate results generated during the animation planning process, including animation scripts, animation spatiotemporal planning knowledge graphs, animation specifications, and final animation result files.

[0022] This invention employs a software-defined approach to animation, decoupling the various actions, movements, rotations, and size modifications of the animated character from the animation material itself and delegating these to software definition. This allows the animation content to be flexibly changed and adjusted as needed. A knowledge-based method is used during animation spatiotemporal planning to schedule the actions of the animated objects, ultimately producing an animation result corresponding to a single sentence in approximately 2 seconds. This satisfies the requirements of real-time performance and ease of use in children's animation generation. It provides a reference example for software-defined real-time animation generation systems. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention;

[0024] Figure 2 This is a structural diagram of the animation knowledge base in this invention;

[0025] Figure 3This is an example diagram of script extraction in this invention;

[0026] Figure 4 This is an example diagram of the qualitative spatiotemporal planning results in this invention;

[0027] Figure 5 This is an example diagram of the quantitative spatiotemporal planning results in this invention;

[0028] Figure 6 This is a system architecture diagram of the present invention;

[0029] Figure 7 This refers to the arrangement rules of certain areas in this invention;

[0030] Figure 8 These are some of the point-taking strategy rules in this invention;

[0031] Figure 9 This refers to the partial motion trajectory splitting rules in this invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] A software-defined method for automatically generating children's animation specifications, the specific implementation steps of which are as follows:

[0034] The first step is to build an animation knowledge base.

[0035] The animation knowledge base includes the knowledge necessary for animation planning and assembly. Its structure is as follows: Figure 2 As shown, it comprises three main parts: a material knowledge base, a layout knowledge base, and a motion knowledge base. The material knowledge base contains information related to animation materials, while the layout and motion knowledge bases contain the information and rules necessary for animation spatiotemporal planning.

[0036] 1) Construct a material knowledge base. The material knowledge base consists of a material name mapping table, a material information table, and a material library, which respectively store material name mappings, default material information, and material files.

[0037] The asset name mapping table is used to establish the mapping relationship between natural language names and assets. It is a many-to-one mapping table structure. The key is a list of names, and the value directly corresponds to a specific asset in the animation asset library. For character and item assets, the key list has only one item. Each element in the list item is a possible asset name, and the value corresponds to the appearance model name of a character or item in the asset library. For example, the key list is ["Mother Sheep", "Sheep"], and the value is "Sheep". For motion assets, an atomic motion corresponds to a specific animation file. This animation file is determined by both the object playing the animation and the motion name. Therefore, the key list has two items. The first item is the appearance name list, and the second item is the motion name list. For example, for the monkey's resting animation, the key list is: [["Monkey", "Monkey"], ["Sleep"], ["Sleeping"], ["Rest"]], and the value is "Monkey_Rest". For background materials, the key list consists of the background name and the time description. Therefore, the key list also has two list items. The first item is the background name list, and the second item is the time description list. For example, the mapping item "Winter Garden" has the key list as: [["Garden", "In the Garden"],["Winter", "Winter", "Winter"]], and the value is "Garden_Winter".

[0038] The material information table stores the necessary default information for material display, including default appearance information for animated characters and items, as well as background area position information. The default appearance information for characters and items includes their default orientation, default size, and material type. The background area position information includes information for each area of ​​each background image and special position information. Each area is approximated by a rectangle, indicating its center position, length, and width; the special position information marks specific locations, such as the coordinates of the area's entry and exit points.

[0039] The asset library stores specific asset files such as backgrounds, animated object appearances, and motion animation files, and is constructed using a non-relational database. The asset library stores both the specific addresses of various assets and the applicable relationships between animated appearance assets and animation files; that is, it records the list of animation files that can be used with a given appearance asset, as well as its default motion asset name.

[0040] 2) Constructing a Placement Knowledge Base. The placement knowledge base stores the placement rules for animated objects in the scene, divided into a placement region rule base and a point-taking strategy rule base. The former determines which region an animated object should be in when it appears in the scene, while the latter determines the point-taking strategy used to obtain its specific location within that region. These correspond to qualitative and quantitative rules, respectively. Both are defined using the SWRL specification, i.e., using reasoning premises (Body) to derive the reasoning result (Head). Constructing the placement region rule base requires creating the following ontology classes to support rule definition: AnimObject class, whose subclasses include Character and Prop; Background class for the animation scene; Region class within the background; Action class for the animation object; and PointFlag class for location markers (PointFlag). The reasoning premises of the rule are the animation object category, the action category, and the location marker category. The reasoning result of the rule is the region where the animated object appears during initialization. The rule will consider the location marker, action type, and animation object category in sequence, and then determine the region where a given animation object is located. For example, if a location identifier is specified, the region is obtained directly based on the identifier; otherwise, it is determined based on the action type. If the action type is also unknown, it is determined based on the category of the animation object. When building the point-finding strategy rule base, a point-finding strategy class, PointStrategy, needs to be created first. Its subclasses include MiddleFirstStrategy, LeftFirstStrategy, RightFirstStrategy, and RandomStrategy. The reasoning premise of the rule includes the category of the animation object, the action category, and the region it is located in. The reasoning result is the point-finding strategy used when the animation object appears. The rule selects the point-finding strategy based on the animation object type, the type of appearance action, and the type of the region. For example, to highlight the importance of character-type animation objects, a "middle-first strategy" is usually used, while a "left-first strategy" is generally used for item-type animation objects. Some layout region rules and point-finding strategy rules are as follows: Figure 7 and Figure 8 As shown.

[0041] 3) Constructing an Action Knowledge Base. The action knowledge base stores information and planning rules for various actions, including an action category mapping table, a compound action splitting table, and an action trajectory splitting rule base. The action category mapping table records which category each action belongs to; the compound action splitting table records which atomic actions a compound action needs to be broken down into; the action trajectory splitting rule base defines the character's movement trajectory after the action is split, also using the SWRL rule format. To categorize various actions and formulate separate trajectory splitting rules, a series of action classes and atomic movement classes are defined. Action classes include: SingleAction, which requires only information from one animation object, and CoupleAction, which requires information from more than one animation object. Based on whether the action causes the animated object to move, it can be further subdivided into SingleMoveAction, SingleStaticAction, CoupleMoveAction, and CoupleStaticAction. Based on whether it's an entrance action, single actions can be divided into entrance actions (ShowupAction) and non-entrance actions (NoShowupAction). The resulting movement trajectory consists of a series of atomic movement classes and their order relationships. Subclasses of atomic movement classes include: FaceToObject, FaceToPointFlag, TurnAround, MovePointFlag, MoveToObject, MoveForward, MoveBackward, and StayStatic. The reasoning premise of the rule is the type of action and the location identifier content. The reasoning result is a number of atomic movement action classes and the order relationship between them. The reasoning result can represent the trajectory of a certain action.

[0042] The action category mapping table and composite action decomposition table in the action knowledge base belong to the qualitative rule level, while the action trajectory decomposition library belongs to the quantitative rule level. Based on the action category mapping table and atomic animation decomposition table, we can determine the category of a given action and, if it is a composite action, which atomic actions it is broken down into. Based on the action trajectory decomposition rule library, we can obtain the movement trajectory of the action. Finally, combined with the current scene conditions, the Dijkstra path planning algorithm is used to calculate the specific numerical value of the trajectory.

[0043] The second step is to correct the speech recognition results input by the child and extract the animation script from the child's natural language text. This step involves combining the information from the material knowledge base to correct the speech recognition results and then extracting the animation script from the corrected text.

[0044] Error correction for speech recognition results includes word segmentation correction, part-of-speech tagging correction, and word content correction. Word segmentation correction aims to extract natural language descriptive words from the source material knowledge base as much as possible from the input text. This is achieved by generating word frequencies based on these descriptive words. First, nouns from the source material database are added to the word frequency file to ensure that all source material names are segmented. Second, if a word's string is a substring of another word, its frequency should be lower. Based on these two principles, a word frequency correction file can be generated from the animation source material knowledge base. Part-of-speech tagging correction involves storing the parts of speech of important words that are prone to missegmentation by the word segmentation model. After part-of-speech tagging, the tagging results are corrected. Word content correction is performed after obtaining the word segmentation results. The process involves identifying mismatched source materials in the source material database that have the same pinyin and replacing the words in the previous text with the names of those source materials.

[0045] Next, we need to extract the animation script from the corrected natural language text. The animation script expresses the time, place, characters, actions, and required props. It's a qualitative description, providing a sequence of all characters and their actions. For easier representation, the animation script is broken down into a sequence of animation script fragments, each describing events occurring within a specific time period. These fragments maintain a strict temporal relationship. Each fragment can be defined as a 4-tuple, i.e., Frag:=<background, propSet, actorSet,actionSet> , respectively representing the background, props, characters, and actions of this segment.

[0046] For real-time feedback, each sentence input by the child is transformed into an executable script fragment. Although script fragments involve many factors, the extraction process prioritizes character actions; other elements, such as background and props, are considered supplementary. Therefore, script fragment extraction focuses on extracting character actions. For an action, a four-tuple is defined as Action:=<subject,action,object,pointFlag> These represent the object of the action, the content of the action, the recipient of the action, and a location marker, respectively. Due to the lack of standardization in natural language expression, some sentences are quite complex, containing multiple actions by multiple roles. Therefore, the difficulty in this step lies in analyzing and understanding complex sentences to obtain the actions of each role and their sequential relationships. The core strategy is to use the predicate verb as the basic unit, breaking down complex sentences into simpler sentences for processing. Based on common Chinese verb predicate sentences, the following four common basic simple sentence types are summarized: subject-predicate sentences, subject-predicate sentences with prepositional phrases and adverbial modifiers, sentences using the "ba" radical, and sentences using the "bei" radical. These simple sentences can all be directly mapped to a single action. For a complex sentence containing multiple predicate verbs input by a child, it is broken down into one of the above four types of simple sentences, using the predicate verb as the unit, according to the structure of the complex sentence, and then specific extraction is performed.

[0047] During sentence segmentation, it's also necessary to complete missing subjects or objects and resolve pronouns. Missing semantics are often filled in by previously encountered semantics; therefore, a semantic exploration algorithm based on the proximity principle is defined. It backtracks by simple sentence unit. If a simple sentence is matched and found to be missing subject or object information, it first explores the subject or object records of neighboring simple sentences. If subject or object information is found, it is substituted into the current simple sentence; otherwise, it continues exploring the previous simple sentence. If no subject is found, it searches the subject or object list of the previous script segment and substitutes it. If no subject information is found, processing of the current simple sentence is abandoned. Based on the list of character actions, descriptions of their background and props can be added; this involves extracting keywords. If background or props are not mentioned, the list can be left blank.

[0048] The third step is animation time-space planning.

[0049] This step involves converting qualitative animation script fragments into quantitative animation information and storing them in the form of a spatiotemporal knowledge graph. The planning process is divided into two main steps: qualitative spatiotemporal planning and quantitative spatiotemporal planning. Qualitative planning involves obtaining the atomic actions of the animation objects, the sequence relationships of these actions, and the areas of activity, ultimately resulting in an animation temporal knowledge graph. This graph contains entities such as animation objects, atomic actions, action locations, and action backgrounds, primarily including the "prior to" and "synchronous" order relationships between atomic actions. Quantitative planning involves obtaining quantitative information such as the appearance location, action trajectory, and execution time of the animation objects, ultimately resulting in an animation planning spatiotemporal knowledge graph. This graph contains entities such as animation objects, atomic actions, time, coordinate values, size values, and orientation, including the start and end relationships between the animation objects and this spatiotemporal information. The main basis for planning is the state of the animation objects in the current scene (represented by the spatiotemporal knowledge graph) and the animation knowledge base constructed in step one.

[0050] 1) Qualitative Planning

[0051] Qualitative planning includes completing the animation script, determining the layout areas, and breaking down complex actions. Completing the script involves adding actions and backgrounds that the child didn't mention in their input but that contribute to the animation's flow. For action completion, the system will use the character's default actions. As mentioned in the resource library description in step one, each character has its preset default actions, such as the rabbit standing and resting, and the turtle swimming. If a character in a script segment lacks a specified action, the system will automatically assign its corresponding default action. The strategy for background completion is to use the background from the previous script segment unless a new background is proposed.

[0052] Determining the layout area means identifying the region where an animated object will appear in the scene. This determination primarily relies on the layout area rule library built in step one, which stores the layout rules for animated objects in various scenarios. Qualitatively, the defined rules consider location identifiers, action types, and the type of animated object in sequence to determine the area where a given animated object will be located. If a location identifier is specified, the area is directly determined by the identifier; otherwise, it is determined based on the action type. If the action type is also unknown, the area is determined based on the category of the animated object.

[0053] Decomposing complex actions refers to breaking them down into atomic actions that can be mapped to specific animation files. An atomic action consists of its corresponding animation and movement trajectory. In qualitative planning, only a list of atomic actions corresponding to the complex action is required. Since different types of actions are decomposed differently, the first step is to query the action category mapping table to obtain the category of a given action. The second step is to query the atomic animation decomposition table to obtain which atomic actions the action corresponds to and the order relationship between these atomic actions. After completing the action decomposition, based on the order relationships within the script, the order relationships between all atomic actions in the script segment can be further obtained.

[0054] 2) Quantitative Programming

[0055] Quantitative programming includes planning the specific placement of animated objects, planning specific motion trajectories, and planning motion execution times. Planning the specific placement involves obtaining the point selection strategy for that area, and combining this with a graphical layout algorithm to obtain the specific position coordinates. The main point selection strategies defined in the rule base are: "center priority strategy," "left priority strategy," "right priority strategy," and "random point selection strategy." The principle for setting the rules is that when the animated object is a character, to highlight the character's importance in the frame, the center priority strategy is chosen in most cases; when the animated object is an item, a left-right priority strategy or a random strategy is adopted depending on its location. Some rules in the rule base are as follows: Figure 7 and Figure 8 As shown. After determining the region and point selection strategy, and considering the actual situation of the current scene, a layout algorithm is used to obtain the specific coordinates of the animated objects. The initial orientation and size data of the animated objects can be obtained by querying the material information table in the material knowledge base constructed in step one.

[0056] Planning a motion trajectory involves obtaining the specific movement path corresponding to the action. When constructing the motion knowledge base, eight atomic movement actions have been defined, and all movement trajectories are composed of these atomic movement actions. After obtaining these atomic movement actions, and combining them with the current specific background and scene state, the Dijkstra path planning algorithm is used to further obtain specific information such as the starting coordinates, ending coordinates, starting orientation, ending orientation, starting size, and ending size of each atomic movement in the action.

[0057] Planning the execution time of actions involves obtaining the total execution time of each character's actions and the execution time of each atomic action, which are related to the time of the child's voice input and the type of action, respectively. To ensure audio-visual synchronization, the total time is defined as the duration of the child's voice input. In planning the execution time of each action, all action execution times are evenly distributed according to the total time. Complex actions are broken down into atomic actions, which are divided into instantaneous actions and continuous actions. Instantaneous atomic actions, such as "turning the head," do not consume time, while continuous actions, such as "walking forward," are evenly distributed according to the total time of the complex action.

[0058] After quantitative programming, an animation spatiotemporal planning knowledge graph can be obtained. This graph stores the state of all animation objects in the scene after each planning step. It serves as the basis for generating animation specifications and as a prerequisite for the next iteration of planning.

[0059] The fourth step is to obtain the specification document that can be used for the final animation display based on the spatiotemporal knowledge graph of the animation planning.

[0060] This step involves combining information from the animation spatiotemporal knowledge graph to generate an animation specification file. Based on the animation display requirements, the animation specification is defined as a sequence of keyframes. Each keyframe refers to an event that occurs within a certain time period, including subtitle generation, sound playback, background transitions, and character animation. It mainly consists of elements such as occurrence time, execution object, prefab (pre-defined motion animation or image), duration, start position, and end position. Specifically, it is a list of keyframes, with each item having the structure: Keyframes:= <action,type,name,goname,timestamp,startPosition,endPosition,startRotation,endRotation,startScale,endScale,duration,content,loop> The `action` field identifies the category of this frame, including the appearance of an animated object, the disappearance of an animated object, playing an animation and moving, switching background images, playing an audio file, and displaying subtitles. `type` identifies the type of animated object being played. `name` and `goname` refer to the name of the source material and the name of the scene object, respectively. `duration` indicates the duration of this frame. The content depends on the frame type; if it's an animated frame, `content` is the name of the animated source material; if it's an audio frame, `content` is the path to the audio file. `loop` indicates whether the animation loops when the frame is an animated frame. Other fields, such as `startPosition`, identify the starting and ending positions, directions, and dimensions of the animated object.

[0061] The method of generating keyframes depends on their category. For keyframes showing the arrangement and actions of animated objects, the start and end times and coordinates of each action for each character can be obtained from the spatiotemporal planning knowledge graph obtained in step three, and keyframes can be generated directly based on this information; audio playback keyframes are generated by playing the audio file stored after the user inputs the audio; and subtitle keyframes are generated based on the text results obtained from speech recognition.

[0062] Fifth step: Assemble the animation according to the animation specifications.

[0063] This step involves retrieving material resources from the animation material library based on the material and spatiotemporal information in the animation specifications, assembling and displaying them on the display platform, including obtaining material information, assembling material files, and calling the material animation interface to play the animation.

[0064] Obtaining material information involves retrieving the appearance material name and motion material name from the animation specification and querying the corresponding material address in the animation material database. Assembling material files involves assembling the appearance material with its playable motion material file. Calling the material animation interface refers to using function calls to manipulate the animation material to play the animation. To enable various types of materials to be automatically invoked according to the animation specification, an interface encapsulating the animation playback function of animation objects is defined, along with a method library for manipulating various materials to perform specific animation functions (playing an animation, moving to a point, rotating by a certain angle, etc.). The parameter format of the interface corresponds to the fields in the keyframes. For example, the core parameter for calling the animation playback keyframe is (matName, actionName, startTime, duration, startPos, endPos, startFace, endFace, startScale, endScale). When different types of materials call the same interface, the underlying system will call different specific execution methods in the method library based on the material type, ultimately presenting the animation, movement, and deformation effects of various animation materials. Example

[0065] Next, taking "Riverside Story" as an example, the animation generation method of this invention will be described in detail. "Riverside Story" consists of four sentences input by the user, the content of which is: (1) By the river, a little pig saw a little rabbit walking in the air. (2) The little pig also walked around on the ground. (3) At this moment, a little turtle swam over. (4) The little pig and the little rabbit greeted the little turtle together. Reference Figure 1 The real-time animation generation method of the present invention mainly includes the following steps:

[0066] The first step is to build an animation knowledge base.

[0067] The animation knowledge base includes the knowledge necessary for animation planning. It consists of three parts: a material knowledge base, a layout knowledge base, and a motion knowledge base.

[0068] 1) Establish an animation asset knowledge base. The animation asset knowledge base includes an asset name mapping table, an asset default information table, and an asset database. The characters involved in "Riverside Story" are "Piglet," "Rabbit," and "Turtle," and their actions include "taking a walk," "walking around," "swimming over," and "greeting," with the setting being "riverside." Next, we will establish corresponding animation asset knowledge bases for these characters and actions.

[0069] Establish a name mapping table. For the mentioned assets, add appearance asset name mappings: [{key:[[“pig”, “pig”]], value: “pig”}, {key:[[“rabbit”, “white rabbit”, “rabbit”]], value: “rabbit”}, {key:[[“turtle”, “turtle”]], value: “turtle”}]; action asset name mappings: [{key:[[“rabbit”], [“walking”]], value: “rabbit_walk”}, {key:[[“rabbit”, [“standing”]], value: “rabbit_idle”}, {key:[[“rabbit”], [“greeting”]], value: “rabbit_speak”}]. Only the rabbit action name mapping is shown here; the other characters follow the same format. Next, create a background name mapping: {key:[[“riverbank”]], value: “riverBank”}. After establishing the name mappings, the natural language descriptions and asset names are associated.

[0070] Create a resource information table. For character and item appearance resources, since the original size and orientation of the resources vary, it is necessary to set their actual size and orientation when displayed in Unity. During setup, adjust the resources to a suitable size in the scene, then set the orientation so that all resources initially face to the right. Record this suitable size and orientation data as the initialization information for the appearance resources. Background resource information mainly consists of area information and marker point information. In the example, three areas—"ground," "river water," and "sky"—are defined for the "riverside" resource, and two types of marker points—"entrance / exit point" and "center point"—are marked for each area.

[0071] Add assets to the animation asset library. The asset files to be added include backgrounds, animation object appearances, and motion assets. Background assets are in general image format; animation object appearance assets are in prefab format; motion assets can be in FBX or anim format. When adding assets, first add the asset name, asset type, and file address for each type of asset. After adding them, you also need to establish the applicability relationship between appearance and motion assets, that is, add a list of animations applicable to a particular appearance asset, and a list of all appearance assets applicable to that animation.

[0072] 2) Establish a layout knowledge base

[0073] The layout knowledge base includes a layout region rule base and a point-taking strategy rule base. Regardless of the specific implementation, it is established according to step 1 in the description. Part of the layout knowledge base is as follows: Figure 7 and Figure 8 As shown.

[0074] 3) Establish a motion knowledge base

[0075] The action knowledge base includes an action category mapping table, a compound action splitting table, and an action trajectory splitting rule base. The construction of the rule base is independent of any specific implementation, but may be expanded as new special actions are added. For example, if "walking around" is a new action, it is first added as a single-person movement action in the action category mapping table, then split into two atomic "walk" actions in the compound action splitting table, and finally, Action_6 is added to the action trajectory splitting rule base. This action consists of three atomic movement actions: MoveForward, TurnAround, and MoveForward, and they are in a sequential order.

[0076] The second step is to correct errors in the speech recognition results input by children and extract the animation script from the children's natural language text.

[0077] 1) Correction of speech recognition results. Taking the first sentence of the "Riverside Story" as an example, the content is "By the river, a little pig saw a little rabbit walking in the air." To ensure that elements related to the material are segmented, the following names have been generated in the word frequency table: background name: "riverside"; character names: "little pig" and "little rabbit"; action names: "saw" and "walked." This ensures that these words are segmented. The second sentence is: "The little pig also walked around on the ground." In the actual speech recognition process, "little pig" may be recognized as the same pronunciation "Xiao Zhu." Since the system recognizes "Xiao Zhu" as a noun or person's name, it will try to find the same pinyin in the material library. It finds that the word "little pig" can be matched, so the system determines that it is a correctable word and corrects it to "little pig."

[0078] 2) Extract the animation script from the corrected natural language text. A script fragment is in the form of a 4-tuple, Frag:=<background, propSet, actorSet, actionSet> The action is in the form of a 4-tuple Action:=<subject,action,object,pointFlag> For the first sentence in the example: "By the river, a little pig saw a little rabbit walking in the air," keyword matching reveals the background as "by the river," the character set as {"little pig," "little rabbit"}, and no items are detected, so the item set is empty. Next is action extraction. Two predicate verbs, "saw" and "walked," are detected, which can be broken down into two simple subject-predicate sentences: "A little pig saw a little rabbit" and "The little rabbit was walking in the air." From the first sentence, the subject of the action "saw" is "little pig," and the object is "little rabbit." From the second sentence, the subject of the action "walked" is "little rabbit," and the location marker is "in the air." Therefore, the action set is {action1{"little pig," "saw," "little rabbit," ""}, action2{"little rabbit," "walked," "," "in the air"}}. Thus, the content of script segment 1 corresponding to the first sentence is:

[0079] frag1{“Riverside”, {}, {“Pig”, “Rabbit”}, {action1{“Pig”, “See”, “Rabbit”, “”}, action2{“Rabbit”, “Walking”, “”, “Air”}}}.

[0080] Following the same logic, the content of script 2 can be derived as follows:

[0081] frag2{“”, {}, {“pig”}, {action1{“pig”, “walking around”, “”, “on the ground”}}}; The content of script 3 is:

[0082] frag3{“”, {}, {“Little Turtle}, {action1{“Little Turtle”, “Swim over”, “”, “”}}}; The content of script 4 is:

[0083] frag4{“”,{},{“pig”,“rabbit”,“turtle”},{action1{“pig”,“greet”,“turtle”,“”},action2{“rabbit”,“greet”,“turtle”,“”}}}.

[0084] The extracted fragment results are as follows Figure 3 As shown in the figure, the sequence relationship of each segment and the actions in the segment are mainly given.

[0085] The third step is to conduct spatiotemporal planning of animation objects based on the animation script and the animation knowledge base, and obtain the spatiotemporal knowledge graph of animation planning.

[0086] 1) Qualitative Planning. This includes completing the animation script, determining the layout areas, and breaking down the actions. Completing the script includes action completion and background completion. As described in step four, actions need to be completed for each character in the script. The system sets default actions for each character, which serve as the basis for completion. In segment 2, the standing action needs to be completed for the rabbit; in segment 3, the standing actions need to be completed for both the rabbit and the pig; and in segment 4, the swimming action needs to be completed for the turtle. Since no further background appears after the background "riverside" is mentioned in the first sentence, the background for segments 2 to 4 of the script is "riverside".

[0087] The primary basis for determining the layout area is the layout rule library in the knowledge base. Based on the information in the script fragment, the area where the action occurs and the point selection strategy are obtained. For example, in script fragment 1, the animated object "pig" is a character class, appearing for the first time in the scene. The corresponding action "seeing" is not an entrance action and has no location identifier. Therefore, according to... Figure 7 Rule 1 in the script assigns it to the default area. Since "Piglet" is a ground character, the default area is the ground. The locations of other actions are determined similarly by querying the rule base from the script fragments to find their respective areas.

[0088] Decomposing an action requires determining its category and the corresponding atomic animations. Category information is used for subsequent quantitative planning of the action's trajectory, while atomic animations are used to map it to one or more specific animation files. For simple actions, such as "taking a walk" in script 1, first, according to the action category mapping table in the action knowledge base, it is determined to be a single-person movement action. Then, according to the atomic animation decomposition table, it is determined to consist of the atomic animation "rabbit_walk". Similarly, for the complex action "walking around" in script fragment 2, first, according to the action category mapping table, it is determined to be a single-person movement action. Then, according to the atomic animation decomposition table, it is determined to consist of two atomic animation segments "pig_walk". After qualitative planning, the result will be as follows: Figure 4 The intermediate planning results are shown.

[0089] 2) Quantitative Planning. This includes planning the specific placement of animated objects, planning specific motion trajectories, and planning motion execution times. Planning the specific placement involves obtaining the point selection strategy, which is then used to calculate the specific placement positions based on the current scene conditions. For characters without any appearance actions, a center-first strategy is typically used to select points to highlight their position in the scene. In the "Riverside Story" example, all characters use a center-first strategy, selecting points in their corresponding areas. After determining the areas and point selection strategies, the system can obtain the state information of each animated character in the scene from the current knowledge graph. Combining this information, the system can use appropriate methods to obtain specific placement coordinates and orientation planning data. For example, if the center-first strategy is used and another animated object is already in the middle of the area, then an empty coordinate will be selected in the adjacent position as the generation position of the new animated object.

[0090] The basis for planning motion trajectories is the motion trajectory decomposition rule base in the motion knowledge base. This is used to obtain the specific trajectory of complex actions. If a certain action cannot be applied to any existing category of decomposition rules, a new rule is introduced for it. For example, according to... Figure 9 According to rule 6, the trajectory of the action "walking around" consists of three atomic movement actions: "MoveForward", "TurnAround" and "MoveFord".

[0091] Planning the execution time of actions includes planning the total execution time of all actions and the execution time of each atomic action. The total execution time of all actions is determined by the recording duration of a sentence. For example, in the story by the river, the recording duration of the second sentence is 2 seconds, so the execution time of the action "walking around" is 2 seconds. Further planning the execution time of atomic actions, the pig's "walking around" is a complex action, which is broken down into three atomic actions: walking forward (corresponding to the animation file pig_walk, atomic movement moveForward), turning head (not corresponding to an animation file, atomic movement turnAround), and walking forward (corresponding to the animation file pig_walk, atomic movement moveForward). Among them, the first and third atomic actions are continuous atomic actions, each occupying 1 second, and the second atomic action is an instantaneous atomic action, which does not occupy time.

[0092] After quantitative programming, the following can be obtained: Figure 5 The spatiotemporal knowledge graph shown, in which, Figure 5 'a' represents the quantitative planning result for the animated object "rabbit"; Figure 5 b represents the quantitative programming result of the animation object "turtle"; 5c represents the quantitative programming result of the animation object "pig".

[0093] The fourth step is to obtain the animation specifications that can be used for the final animation display based on the animation planning spatiotemporal knowledge graph.

[0094] The animation specification file includes all the necessary control frames for the animation, such as audio, subtitles, background, and motion. For audio frames, the action type is 3, which needs to include the audio file name, start time, and duration, as follows: {"action": 3,"type": 2,"name": "20228917737_story0","goname": "","timestamp": 0.0,"startpos": [ 0.0, 0.0, 0.0 ],"endpos": [ 0.0, 0.0, 0.0 ],"startrotation": [ 0.0, 0.0, 0.0 ],"endrotation": [ 0.0, 0.0, 0.0 ],"startscale": [ 0.0, 0.0, 0.0 ],"endscale": [ 0.0, 0.0, 0.0 ],"duration":4.76,"content": "","loop": 0};

[0095] The subtitle display frame has an action type of 5 and needs to include the subtitle content, start time, and duration, as follows: {"action": 5,"type": 2,"name": "","goname": "1","timestamp": 0.0,"startpos":[ 0.0, 0.0, 0.0 ],"endpos": [ 0.0, 0.0, 0.0 ],"startrotation": [ 0.0, 0.0,0.0 ],"endrotation": [ 0.0, 0.0, 0.0 ],"startscale": [ 0.0, 0.0, 0.0 ],"endscale": [ 0.0, 0.0, 0.0 ],"duration": 4.76,"content": "In the forest, a little pig saw a little rabbit walking in the air","loop": 0}.

[0096] The remaining keyframes are generated based on the content in the current spatiotemporal knowledge graph. For background transition frames, the action type is 4, which needs to include the background file name, start time, and duration; for character initialization frames, the action type is 0, which needs to include the appearance asset name, start time, duration, start coordinates, end coordinates, start orientation, end orientation, start size, and end size; for character action frames, the action type is 2, which includes the appearance asset name, animation asset name, whether to loop, start time, duration, start coordinates, end coordinates, start orientation, end orientation, start size, and end size.

[0097] Fifth step: Assemble the animation according to the animation specifications.

[0098] This step involves retrieving content from the animation asset library based on the asset and spatiotemporal information in the animation specification, and then assembling and displaying it in real-time within Unity. First, based on the asset names in the animation specification, the asset library is queried to obtain the appearance assets and related motion assets for "pig," "rabbit," and "turtle." Next, the motion assets corresponding to the appearance assets are assembled in Unity, resulting in a list of motion assets assembled for all appearance assets. Finally, these motion assets are sorted according to the start time of the keyframes and displayed sequentially, using coroutines to ensure that the display of each keyframe does not interfere with others. For the animation object's appearance keyframe (action=0), the corresponding material is retrieved based on the "name" field in the keyframe. After adjusting its position, orientation, and size information according to the keyframe, it is placed on the canvas. For the animation object's action keyframe (action=2), the appearance of the corresponding material is determined based on the "name" field, and the corresponding action material is determined based on the "content" field. Then, based on the action's start and end positions, start and end orientations, start and end sizes specified in the keyframe, the animation playback methods for various materials are called through a unified API to play the animation and perform movement operations. For other keyframes that do not involve calling materials, the relevant Unity interfaces are directly called, and coroutines are used to control the start and duration to display the content in the keyframe.

[0099] Figure 6The system architecture diagram of this invention is shown below. The system usage flow of this invention is as follows: After entering the animation creation interface, using the functions provided by the voice and touch interaction modules in the interaction layer, clicking the record button will start recording the story. Clicking it again will end the recording of the current sentence, and the recording result will be handed over to the function layer. First, the voice recognition and error correction module recognizes and corrects the text. Then, the animation planning module starts to execute the planning based on the voice text and the knowledge base in the knowledge layer to obtain the animation specification file, which is then passed to the animation assembly module. When this module reads the newly generated specification file, it will send a request to the animation material server to query the address information and assembly information of various materials in the specification file, and assemble and display them in Unity. The corresponding animation will be displayed on the canvas.

Claims

1. A software-defined real-time automatic generation method for children's animation, characterized in that, Includes the following steps: Step 1: Build an animation knowledge base; Step 2: Correct the speech recognition results input by the child and extract the animation script from the child's natural language text; Step 3: Based on the animation script and combined with the animation knowledge base, perform spatiotemporal planning of the animation objects to obtain the animation planning spatiotemporal knowledge graph; Step 4: Obtain the animation specification for the final animation display based on the animation planning spatiotemporal knowledge graph; Step 5: Assemble the animation according to the animation specifications; where: Step one specifically involves: constructing the necessary knowledge base for animation planning, including a material knowledge base, a layout knowledge base, and a motion knowledge base; wherein, the material knowledge base includes the mapping relationship between materials and names (i.e., the material name mapping table), the material category, size, and orientation information (i.e., the material information table), and the specific materials (i.e., the material library); the layout knowledge base includes a layout area rule library and a point selection strategy rule library; the motion knowledge base includes the mapping relationship between actions and categories (i.e., the action category mapping table), the mapping relationship between compound actions and atomic actions (i.e., the compound action splitting table), and the action trajectory splitting rules; and selecting a point selection strategy based on the animation object type, the type of appearance action, and the type of the area in which it is located. Step two specifically involves: correcting the speech recognition results input by the child, which means correcting the speech recognition text in three aspects: word segmentation, part of speech, and word content, based on the content of the material library; extracting the animation script from the child's natural language text, which means breaking the corrected text into short sentences based on predicate verbs, and then extracting background, character, object, action sequence, and location marker information from each short sentence based on basic sentence patterns to generate the corresponding animation script segment; Step three specifically involves: based on the animation script obtained in step two, and combined with the animation knowledge base, performing spatiotemporal planning on the animation objects to obtain quantitative information on the action execution time and execution trajectory of each animation object in the current script, and using the quantitative information to generate an animation planning spatiotemporal knowledge graph; defining the total time as the duration of the child's input voice; and dividing all action execution times equally according to the total time; the animation planning spatiotemporal knowledge graph saves the state of all animation objects in the scene after each planning, which serves as the basis for generating animation specifications and as a prerequisite for the next iteration of planning; The fourth step is as follows: Based on the quantitative information of the animation objects in the animation planning spatiotemporal knowledge graph in the third step, and combined with the voice and subtitle content in the scene, generate the animation specification file corresponding to the current input sentence; after the user inputs all sentences, merge the specification files corresponding to each sentence to obtain the specification file for displaying the total animation; define the animation specification file as a keyframe sequence. Step five specifically involves retrieving background, animation object appearance, and motion animation materials from the animation material library based on the material information and spatiotemporal information in the animation specifications, and assembling and displaying them in Unity.

2. A system for automatically generating a child animation in real time, which implements the method according to claim 1, characterized in that, The system comprises: an interaction layer, a functional layer, a knowledge layer, and a data persistence layer; among which: The interaction layer is responsible for handling the interaction between children and the system, including a voice interaction module, a touch interaction module, a material display module, and an animation display module. The voice interaction module is responsible for acquiring the child's voice information; the touch interaction module is responsible for acquiring the child's click and swipe touch operations; the material display module is responsible for displaying the materials in the material library to the child in the form of thumbnails; and the animation display module will play and display the generated animation to the child. The functional layer handles the automatic generation of animations, including a script extraction module, an animation planning module, and an animation assembly module. The script extraction module is responsible for speech recognition and error correction, extraction of entities and entity relationships, and extraction of action sequences. The animation planning module is responsible for planning the time, actions, arrangement, and path elements of each animation object in the animation, and generating animation specifications. The animation assembly module is responsible for finding and calling animation materials, generating subtitles and voice-over, and obtaining complete animation content. The knowledge layer stores the knowledge base needed for animation generation, including the material knowledge base, the layout knowledge base, and the motion knowledge base; The data persistence layer stores various intermediate results generated during the animation planning process, including animation scripts, animation spatiotemporal planning knowledge graphs, animation specifications, and final animation result files.

Citation Information

Patent Citations

  • Fast animation scene construction method and device

    CN104992461A

  • Animation production method and device, terminal and readable medium

    CN108847246A

  • System for generating animations according to natural language based on body cognition

    CN110083818A

  • System for automatically generating animation scene

    CN114022596A