Animation picture book generation method, device, equipment and storage medium

By identifying objects and text in static picture books, generating voice data and picture changing content, real-time conversion of static picture books into dynamic picture books is solved, and the problem of high cost and poor vividness of dynamic picture books in the existing technology is improved, and the vividness of picture books is improved Sex and flexibility.

CN114359446BActive Publication Date: 2024-11-01IFLYTEK CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202111658559.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-30
Publication Date
2024-11-01
Estimated Expiration
2041-12-30

AI Technical Summary

Technical Problem

The video resources of existing dynamic picture books are costly and have low vividness, so it is impossible to effectively convert static picture books into vivid animations.

Method used

By identifying objects and text in static picture books, determining the correspondence between objects and text, generating voice data and picture changing content, combining voice data and target part images, generating animated picture book data, and realizing real-time conversion of static picture books into dynamic picture books .

Benefits of technology

It reduces the configuration cost of dynamic picture books, improves the flexibility and vividness of dynamic picture books, and can transform static characters into animation characters that can be said to be dynamic.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114359446B_ABST
    Figure CN114359446B_ABST
Patent Text Reader

Abstract

The present application discloses a method, device, equipment and storage medium for generating an animated picture book, the method comprising: identifying several objects and several texts in a static picture book; determining the correspondence between each object and each text, wherein the correspondence indicates that the text is narrated by the object corresponding to the text; generating voice data of each text, and determining the screen change content of the object corresponding to each text, wherein the screen change content indicates the change of the preset part of the object during the playback of the voice data corresponding to the text; based on the voice data and the screen change content, generating the animated picture book data corresponding to the static picture book. In the above manner, the present application can convert the static characters in the picture book into animated characters that can speak and move.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for generating an animated picture book. Background Art

[0002] A picture book is a book that mainly consists of pictures and can be connected into a complete story. It is mainly aimed at children. It can help children understand the world, learn knowledge, and cultivate children's interest and hobby in reading.

[0003] The existing picture books on the market can be divided into two categories. One is static picture books, which rely on manual reading or mechanical reading by picture book robots; the other is dynamic picture books, which are equipped with some video resources. The video resources are pre-made and users can watch them by scanning the QR code on the picture book or entering the link in the browser interface. The video resources only make some twists on the static frames of the picture book, which are not very watchable and are expensive. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a method, device, equipment and storage medium for generating an animated picture book, which can generate vivid animated characters.

[0005] To solve the above technical problems, the first aspect of the present application provides a method for generating an animated picture book, the method comprising: identifying a number of objects and a number of texts in a static picture book; determining the correspondence between each object and each text, wherein the correspondence indicates that the text is narrated by the object corresponding to the text; generating voice data for each text, and determining the screen change content of the object corresponding to each text, wherein the screen change content indicates the change of a preset position of the object during the process of playing the voice data corresponding to the text; and generating animated picture book data corresponding to the static picture book based on the voice data and the screen change content.

[0006] To solve the above technical problems, the second aspect of the present application provides an animated picture book generation device, which includes: an identification module, which is used to identify several objects and several texts in a static picture book; a determination module, which determines the correspondence between each object and each text, wherein the correspondence indicates that the text is narrated by the object corresponding to the text; a synthesis module, which is used to generate voice data for each text, and determine the screen change content of the object corresponding to each text, wherein the screen change content indicates the change of a preset part of the object during the process of playing the voice data corresponding to the text; and a generation module, which is used to generate animated picture book data corresponding to the static picture book based on the voice data and the screen change content.

[0007] To solve the above technical problems, the third aspect of the present application provides an animation picture book matching device, which includes a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the method described in the first aspect above.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed to implement the method described in the first aspect above.

[0009] The beneficial effects of the present application are as follows: Different from the prior art, the present application identifies several objects and several texts in a static picture book, determines the correspondence between each object and each text, and converts several texts into voice data, so that the text content in the static picture book can be expressed by voice; further determines the picture change content of the object corresponding to each text, that is, determines the target part image of the object after voice drive, combines the voice data and the target part image, and generates the animation picture book data corresponding to the static picture book. That is, it is possible to convert a static picture book into a dynamic picture book in real time without pre-configuring the dynamic picture book, reducing the configuration cost, and all existing static picture books can be converted into dynamic picture books, improving the flexibility of the dynamic picture book. In addition, the static characters in the picture book are converted into animated characters that can speak and move, improving the vividness of the picture book. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of the first implementation method of the animation picture book generation method of the present application;

[0011] Figure 2 This is a flowchart of an implementation of step S120 in the first implementation of the method for generating an animated picture book of the present application;

[0012] Figure 3 This is a flowchart of an implementation of step S130 in the first implementation of the method for generating an animated picture book of the present application;

[0013] Figure 4 This is a flowchart of another implementation of step S130 in the first implementation of the method for generating an animated picture book of the present application;

[0014] Figure 5 is a flowchart of an implementation method of determining the screen change content of the target object in step S432;

[0015] Figure 6 It is a flowchart of the second implementation method of the animation picture book generation method of the present application;

[0016] Figure 7It is a flowchart of the third implementation method of the animation picture book generation method of the present application;

[0017] Figure 8 It is a flowchart of the fourth implementation method of the animation picture book generation method of the present application;

[0018] Fig. 9 It is a schematic diagram of a framework of an implementation method of an animation picture book generating device provided by the present application;

[0019] Fig.10 It is a schematic diagram of a framework of an implementation method of an animation picture book generating device provided by the present application;

[0020] Fig.11 It is a schematic diagram of a framework of an implementation scheme of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] It should be noted that in the embodiments of the present application, there are descriptions involving "first", "second", etc., which are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0023] See also Figure 1 , Figure 1 1 is a flow chart of a first embodiment of the method for generating an animated picture book of the present application, the method comprising:

[0024] S110: Recognize a plurality of objects and a plurality of texts in a static picture book.

[0025] Among them, several objects are the contents other than text content and picture book background that appear in the static picture book, and may be characters, animals, etc. that appear in the picture book. A target detector may be used to detect the static picture book, obtain the objects contained in the picture book, and mark them. For example, if two character objects are detected, they may be marked as character 1 and character 2. In one embodiment, an animated picture book detector may be used to detect several objects in a static picture book. In other embodiments, other detection devices may also be used, which are not limited here.

[0026] For some texts in the static picture book, in one implementation, OCR (Optical Character Recognition) may be used to recognize and obtain the text content.

[0027] S120: Determine the correspondence between each object and each text.

[0028] In one embodiment, the correspondence relationship indicates that the text is narrated by the object corresponding to the text. Step S110 sends the several objects detected by the detection device and the several text contents recognized to the neural network model. The neural network model encodes the several objects and the several text contents, obtains the image features of each object and the first text features of each text, and obtains the correspondence between each object and each text by calculating the similarity between the image features of each object and the first text features of each text, that is, the narration object corresponding to each text can be obtained. Among them, the neural network model can be an overall model composed of multiple neural networks; it can also be multiple small neural network models, and the correspondence between each object and each text is obtained based on the interaction between them. For example, the image features of several objects in a static picture book are obtained by the first extraction network, the first text features of each text are obtained by the second extraction network, and then the similarity between the image features of each object and the first text features of each text is obtained by the relationship recognition network, and then the similarity between the image features of each object and the first text features of each text is obtained.

[0029] S130: Generate voice data for each text, and determine the screen change content of the object corresponding to each text.

[0030] Specifically, voice data can be generated based on the recognized text content. In one embodiment, a voice synthesis framework (such as tacotron or tacotron2) can be used to synthesize voice data. Furthermore, the screen change content generated by the object in the process of telling the synthesized voice data can be determined, wherein the screen change content represents the change of the preset part of the object in the process of playing the voice data corresponding to the text. The preset part can be the face, body part, etc. of the object. The change of the preset part can be the target part image obtained after playing the voice data corresponding to the text.

[0031] In one embodiment, the preset part may be the face of the object, and further, a number of key points may be defined on the face of the object, and the picture change content may be determined based on the position change of the key points of the object in the process of speaking the voice relative to the key points in the static picture book. In another embodiment, the picture change content may also be determined based on the change of the pixel points on the face of the object.

[0032] S140: Generate animation picture book data corresponding to the static picture book based on the voice data and the picture change content.

[0033] The animated picture book data may be an animation composed of dynamic changes of objects in a static picture book during the narration of synthesized voice data.

[0034] Through the above method, the present application can automatically identify several objects and several texts in the picture book, determine the corresponding relationship between each object and each text, and convert several texts into voice data, so that the text content in the static picture book can be expressed by voice; further determine the picture change content of the object corresponding to each text, that is, determine the target part image of the object after voice drive, combine the voice data and the target part image, and generate the animation picture book data corresponding to the static picture book. That is, it can realize the real-time conversion of static picture books into dynamic picture books, without the need to pre-configure dynamic picture books, reducing configuration costs, and all existing static picture books can be converted into dynamic picture books, improving the flexibility of dynamic picture books, and in addition, the static characters in the picture books are converted into animated characters that can speak and move, improving the vividness of the picture books.

[0035] See also Figure 2 , Figure 2 This is a flow chart of step S120 in the first embodiment of the method for generating an animated picture book of the present application. Step S120 includes:

[0036] S221: using the first extraction network to extract image features of each object from the static picture book; and for each text, using the second extraction network to extract a number of word vectors of the text from the text, and aggregating the several word vectors of the text to obtain a first text feature of the text.

[0037] In one embodiment, for several objects in a static picture book, a neural network (such as a first extraction network) can be used to extract the image features of the object. In a specific embodiment, the first extraction network can be a convolutional neural network (CNN), which uses a convolutional neural network to extract the image features of each object; in another specific embodiment, the first extraction network can be a transformer structure, which uses a transformer structure to extract the image features of each object. For several texts in a static picture book, after being recognized by OCR, the recognized text content can be sent to the second extraction network model, and the second extraction network model extracts several word vectors in the text, and aggregates several word vectors of the text to obtain the first text feature of the text, wherein the first text feature can be obtained by averaging several word vectors. In a specific embodiment, the second extraction network model can be BERT (Bidirectional Encoder Representation from Transformers), which is a pre-trained language representation model; in another embodiment, the second extraction network model can be a GPT (Generative Pre-Training) model, and the GPT-3 model can be specifically used. It can be understood that the GPT-2 model can also be used, which is not limited here.

[0038] S222: Using a relationship recognition network to discriminate the image features of each object and the first text features of each text, and obtain the similarity between each text and each object.

[0039] After obtaining the image features of several objects and the first text features of several texts in the static picture book in step S221, the similarity between each text and each object can be further determined. In one embodiment, the image features of each object and the first text features of each text can be determined using a relationship recognition network to obtain the similarity between each text and each object. The relationship recognition network can be a dual-stream transformer structure.

[0040] S223: Based on the similarity, the correspondence between each object and each text is obtained.

[0041] The similarity between each text and each object is further determined, and the correspondence between each object and each text is determined, that is, which object narrates a certain sentence.

[0042] See also Figure 3 , Figure 3 This is a flow chart of step S130 in the first embodiment of the method for generating an animated picture book of the present application; this step includes:

[0043] S331: Determine the sound reference information corresponding to each text.

[0044] In one embodiment, the sound reference information includes at least one of a role category and an emotion category. The sound reference information corresponding to each text indicates which role narrates the text or what emotion the text expresses. The role category may be male, female, little boy, little girl, etc.; the emotion category may be surprise, happiness, sadness, neutral, etc.

[0045] The sound reference information corresponding to each text can be obtained from a pre-trained model. Specifically, when training the model, the user can pre-set some role categories and emotion categories, and label the role categories and emotion categories; further based on the second text features and the entity features of the named entity, the role category features and emotion category features corresponding to the text are obtained, and it is determined whether they are consistent with the annotations; among them, the named entity is the content in the text, and the named entity is a person's name, an organization name, a place name, and all other entities identified by a name, such as dad, baby, rabbit, etc.

[0046] In actual application, at least one entity feature of a named entity can be extracted from a number of texts. Specifically, the entity feature of the named entity can be directly extracted from the first text feature, which can be obtained by BERT or GPT; or the named entity can be input into the feature model to obtain the entity feature of the named entity.

[0047] The information recognition network is further used to process the second text features of each text and the entity features of each named entity to obtain the sound reference information corresponding to each text. In one embodiment, the information recognition network can be a dual-stream transformer structure, and the corresponding relationship between the second text features of each text and the entity features of each named entity is obtained by the dual-stream transformer structure, and the sound reference information corresponding to each text is determined based on the corresponding relationship. Among them, the second text feature is obtained by average pooling of the word vectors contained in the text, and can be the same as the first text feature or different from the first text feature.

[0048] In another embodiment, the entity features of the named entities can be input into the first transformer structure to obtain the entity relationships between the named entities, and the second text features of the texts can be input into the second transformer structure to obtain the text relationships between the texts. Further, based on the entity relationship and the text relationship, a cross-category attention model can be used to obtain the correspondence between the named entities and the texts. Finally, the corresponding relationship between the named entities and the texts is used to determine the sound reference information corresponding to the text, that is, which character tells a sentence and the emotion expressed by the sentence.

[0049] S332: Generate voice data for each text based on the sound reference information.

[0050] In one embodiment, step S331 obtains the sound reference information of the text including the role category and the emotion category, and determines the target timbre corresponding to the text based on the role category. For example, if the role category corresponding to the text is determined to be a little girl, the characteristics of the girl's voice are obtained; the characteristics of the girl's voice and the emotion category characteristics are added to the speech synthesis framework (such as tacotron2) to obtain speech data rich in emotion. Among them, the characteristics of the girl's voice and the emotion category characteristics can be obtained by a pre-trained model. In another embodiment, the target tone can be determined based on the role category and the emotion category. For example, the text content is "Under our chicken coop, there is a big fire dragon living." Step S331 obtains that the role category corresponding to the text is a little girl, and the corresponding emotion category is surprise. Then, the girl's voice and the surprised timbre can be selected to convert the text into emotion-rich speech data.

[0051] It can be understood that the first extraction network, the second extraction network, the relationship identification network, etc. included in this embodiment can be executed by different modules in one model or by multiple models.

[0052] See also Figure 4 , Figure 4 This is a flow chart of another implementation of step S130 in the first implementation of the method for generating an animated picture book of the present application. This step may also include:

[0053] S431: Detecting a part area corresponding to a preset part of each object in the static picture book, extracting key points from the part area of ​​the object, and obtaining reference key point position information of the object.

[0054] In one embodiment, the preset part includes the face, and the part area corresponding to the preset part is the entire face area, and the key points of the face area are extracted to obtain the position information of the key points, which is the reference key point position information of the object. The reference key point position information includes the position information of multiple key points in the preset part of the object in the static picture book.

[0055] Specifically, a part detector (such as a face detector) can be used to detect facial key points of objects contained in a static picture book. Before application, the face detector can be trained, and a large number of picture books can be collected, so that the face detector detects the face frames of objects contained in a large number of picture books, and the face-of-art method is used to extract the key points of the object according to the face frame. In a specific implementation, the object is a person in a static picture book, and the preset part is a face. The face detector can be used to detect the face frame of the face, extract the key points in the face frames of several people, and obtain the reference key point position information of several people.

[0056] S432: For each text, the object and voice data corresponding to the text are used as the target object and target voice. Based on the reference key point position information of the target object and the target voice, the position changes of multiple key points of the target object during the playback of the target voice are obtained. Based on the corresponding position changes of the target object, the picture change content of the target object is determined.

[0057] In one embodiment, the position change corresponding to the target object includes several groups of key point position offsets of the target object, different groups of key point position offsets correspond to different time segments of the target speech, and each group of key point position offsets includes position offsets of multiple key points of the target object corresponding to the time segment. Specifically, a segment of target speech can be divided into multiple time segments, each time segment can be a speech frame or multiple speech frames, and a time segment can correspond to a group of key point position offsets.

[0058] In a specific implementation, the target object may be a person, and a face detector is used to detect the face frame of the target object in a static picture book, and a number of reference key points are extracted to obtain the reference key point position information. A speech feature extraction model is used to extract the speech features of the target speech, and the speech features may be MFCC (Mel-scale Frequency Cepstral Coefficients) features or deepspeech features.

[0059] Furthermore, the reference key point position information, speech features, and reference factors of the target object are input into the position prediction network to obtain several groups of key point position offsets of the target object. The reference factors include noise and the encoding features of the emotional category of the text corresponding to the target object. The noise represents the head movement and the facial expressions in the preset parts when the target object speaks. Different emotional categories correspond to different motion templates of the preset parts. Different motion templates represent the morphology of the preset parts under a certain emotional category. For example, when the emotional category is surprise and the preset part is the face, the eyes and mouth in the face are wide open.

[0060] In one embodiment, the target speech can be divided into different time segments, and the speech features of a time segment, multiple reference key point position information, noise, and coding features of the emotion category are input into the position prediction network to obtain the offsets of multiple key points of the target object relative to the reference key point position information after playing the target speech of a time segment. By inputting all the time segments contained in the target speech, multiple groups of key point position offsets can be obtained. Among them, the reference key point position information can be the position of the key point of the target object in the static picture book. The position prediction network can be a GRU (Gate Recurrent Unit) in a recurrent neural network, or an LSTM (Long-Short Term Memory), or a transformer network.

[0061] In one embodiment, the position prediction network can use LSTM, and two fully connected layers are added after the sequence layer, representing the position offset and the emotion category respectively. In the actual application process, it is possible to detect whether the encoding features of the emotion category of the input position prediction network are consistent with the emotion category of the output neural network, and make a brief judgment. Before the actual application, the position prediction network can be trained. During the training process, the output emotion category can be strongly decoupled from the input noise and the encoding features of the emotion category, so that the noise is only related to the head movement and eye opening and closing of the object, and the encoding features of the emotion category are related to the emotional state expressed.

[0062] See also Figure 5 , Figure 5 is a flowchart of an implementation method of determining the screen change content of the target object in step S432; step S432 may also include:

[0063] S5321: Based on the reference key point position information of the target object and the key point position offset of the corresponding time segment, obtain the region information of the target object corresponding to the time segment.

[0064] In one embodiment, the reference key point position information of the target object may be the position information of the reference key point of the target object in a static picture book, and the speech data is divided into multiple time segments. After the target object speaks the target speech of a time segment, the position information of the key point will change, and the region information of the target object corresponding to the time segment is obtained by calculating the position offset of the key point after a time segment. In one specific embodiment, the region information includes the position offset of at least some pixels in the part area of ​​the target object; in another specific embodiment, the region information includes the position offset of at least some pixels in the part area of ​​the target object and a region mask, and the region mask indicates the position of a preset region in the target part image, and the preset region is invisible in the corresponding reference part image and visible in the target part image. For example, the preset region may be a tooth region, the object in the reference part image may be in a state of a closed mouth, and the object in the target part image may be in a state of an open mouth.

[0065] S5322: Obtain a target part image of the target object corresponding to a time segment based on the position offset of at least part of the pixels and a reference part image of the target object.

[0066] In one embodiment, the reference part image includes a preset part of the target object, which may be the face. The reference part image is deformed using the position offset of at least some pixels to generate a deformed image of the target object. The deformed image generated at this time may have some regions that are not correct, such as the eye region. Therefore, the deformed image, the feature information of the reference part image, and the position offset of the key points of the corresponding time segment can be further used to obtain the target part image of the target object corresponding to the time segment. The target part image includes the preset part, and several frames of target part images of the target object corresponding to different time segments constitute the picture change content of the target object. The feature information of the reference part image can be some features extracted from the preset region of the object in the static picture book (such as features of the eyes, mouth, etc.), and the feature information of the reference part image can be extracted using a convolutional neural network.

[0067] Furthermore, after obtaining the target part image, the preset area can be filled with content, wherein the preset area is invisible in the corresponding reference part image and visible in the target part image, and the preset area can be teeth, eyes and other areas.

[0068] See also Figure 6 , Figure 6 It is a flow chart of the second implementation method of the animation picture book generation method of the present application; the method comprises:

[0069] S610: Recognize a number of objects and a number of texts in a static picture book.

[0070] S620: Determine the correspondence between each object and each text.

[0071] Among them, the correspondence relationship means that the text is told by the object corresponding to the text.

[0072] S630: Generate voice data for each text, and determine the screen change content of the object corresponding to each text.

[0073] The image change content indicates the change of the preset part of the object during the playback of the voice data corresponding to the text. In one embodiment, the image change content of the object includes several frames of target part images of the object with respect to the preset part, and different frames of target part images correspond to different time segments of the voice data.

[0074] S640: For each time segment, the background image of the static picture book is spliced ​​with the target part image of the corresponding time segment to obtain a picture book frame of the corresponding time segment.

[0075] In one embodiment, a plurality of reference key points may be marked in a preset part of an object of a detected static picture book, wherein the preset part may be a face. A plurality of background points are expanded in a background area outside the face, and a first positional relationship between each reference key point and each background point is determined. Before obtaining the target part image of each time segment, the offset of the key point in the target part image relative to each reference key point may be obtained. After obtaining the target part image of each time segment, the background image in the picture book may be dragged based on the offset, so that the background image of the static picture book is spliced ​​with the target part image of the corresponding time segment, ensuring that the second positional relationship between the background points in the spliced ​​background image and the key points in the target part image is the same as the first positional relationship.

[0076] S650: Generate animation picture book data based on the voice data and a plurality of picture book frames corresponding to different time segments of the voice data.

[0077] In one embodiment, several picture book frames may be integrated to obtain a video, and the voice data and the obtained image may be further combined to generate animated picture book data. In another embodiment, the voices of each time segment and the picture book frames corresponding to the decibels may be combined to obtain animated picture book data of a time segment, and the animated picture book data of multiple time segments may be further combined to generate complete animated picture book data.

[0078] See also Figure 7 , Figure 7 It is a flowchart of the third embodiment of the method for generating an animated picture book of the present application; the method comprises:

[0079] S710: Recognize a number of objects and a number of texts in a static picture book.

[0080] S720: Determine the correspondence between each object and each text.

[0081] Among them, the correspondence relationship means that the text is told by the object corresponding to the text.

[0082] S730: Generate voice data for each text, and determine the screen change content of the object corresponding to each text.

[0083] The picture change content indicates the change of the preset position of the object during the process of playing the voice data corresponding to the text.

[0084] For the above steps S710-S730, please refer to steps S110-S130 in the first embodiment of the animation picture book generation method, which will not be repeated here.

[0085] S740: Obtaining the positional relationship between a plurality of key points of a preset part of the object and a preset number of associated background points.

[0086] In one embodiment, the preset part can be a face, and multiple key points are marked in the face area of ​​the object, and a preset number of background points are expanded in the area outside the face as associated background points to determine the positional relationship between the key points of the face and the associated background points. Among them, the associated background points are located within a preset distance from the boundary of the preset part of the object in the static picture book; the preset number can be 10, 20, or 30, which is not limited here. In a specific embodiment, the positional relationship between the key points of the face and the associated background points can be determined by establishing a Dirichlet triangle, and multiple Dirichlet triangles can be established so that each triangle contains at least one key point of the face and an associated background point.

[0087] S750: For each time segment, the background image of the static picture book is spliced ​​with the target part image of the corresponding time segment to obtain a picture book frame of the corresponding time segment.

[0088] S760: Adjusting the positions of the associated background points in the picture book frame based on the positional relationship of the objects.

[0089] In one embodiment, based on the Dirichlet triangle established in step S740, the associated background points are pulled to move so that the positional relationship between the key points of the target part image in the picture book frame and the associated background points is the same as the positional relationship between the key points of the object in the static picture book and the associated background points.

[0090] In this embodiment, by determining the relationship between the key points and the associated background points, the background image and the target part image are perfectly combined, thereby making the generated animation vivid and natural.

[0091] See also Figure 8 , Figure 8: is a flowchart of the fourth embodiment of the method for generating an animated picture book of the present application; the method comprises:

[0092] S810: Recognize a number of objects and a number of texts in a static picture book.

[0093] S820: Determine the correspondence between each object and each text, wherein the correspondence indicates that the text is narrated by the object corresponding to the text.

[0094] S830: Generate voice data for each text, and determine the screen change content of the object corresponding to each text.

[0095] S840: Generate animation picture book data corresponding to the static picture book based on the voice data and the picture change content.

[0096] For the above steps S810-S840, please refer to steps S110-S140 of the first embodiment of the animation picture book generation method, which will not be repeated here.

[0097] S850: Select at least one object from the static picture book as a message object.

[0098] After generating the animation picture book data, at least one object in the static picture book can also be selected to communicate with the reader. In one embodiment, the communication object can be selected according to the size of the object frame where the object is located detected in step S810. In another embodiment, the communication object can be randomly determined.

[0099] S860: Based on the preset content and / or the text content in the static picture book, drive the dialogue object to have a dialogue with the user.

[0100] In one embodiment, if the scanned static picture book is a story picture book, an object with a larger object box detected in the story picture book can be selected as the message object, and a certain time segment can be selected to play the picture book animation data corresponding to the time segment to communicate with the reader; the reader can also be asked questions based on preset content, such as a simple question.

[0101] In another embodiment, the scanned static picture book is a language picture book (such as English, Pinyin), and voice data of a time segment can be randomly selected and played to allow the reader to follow and learn, thereby guiding the reader's pronunciation.

[0102] This implementation method increases interaction with readers by introducing a simple sentence or a short story, thereby improving readers' interest in learning.

[0103] See also Fig. 9 , Fig. 9It is a framework diagram of an embodiment of an animation picture book generation device provided by the present application. The animation picture book generation device 90 includes: an identification module 91, a determination module 92, a synthesis module 93, and a generation module 94. The identification module 91 is used to identify a number of objects and a number of texts in a static picture book; the determination module 92 determines the correspondence between each object and each text, wherein the correspondence indicates that the text is narrated by the object corresponding to the text; the synthesis module 93 is used to generate voice data for each text, and determine the screen change content of the object corresponding to each text, wherein the screen change content indicates the change of the preset part of the object during the process of playing the voice data corresponding to the text; the generation module 94 is used to generate the animation picture book data corresponding to the static picture book based on the voice data and the screen change content.

[0104] Among them, identifying a number of objects and a number of texts in a static picture book includes: performing object detection on the static picture book to obtain a detection result containing a number of objects; and performing text recognition on the static picture book to obtain a number of texts.

[0105] Among them, determining the correspondence between each object and each text includes: using a first extraction network to extract image features of each object from a static picture book; and, for each text, using a second extraction network to extract a number of word vectors of the text from the text, and aggregating the several word vectors of the text to obtain a first text feature of the text; using a relationship recognition network to distinguish the image features of each object and the first text features of each text, and obtain the similarity between each text and each object; based on the similarity, obtaining the correspondence between each object and each text.

[0106] Generating the voice data of each text includes: determining the sound reference information corresponding to each text, wherein the sound reference information includes at least one of a role category and an emotion category; and generating the voice data of each text based on the sound reference information.

[0107] Among them, determining the sound reference information corresponding to each text includes: extracting entity features of at least one named entity from a number of texts; using an information recognition network to process the second text features of each text and the entity features of each named entity to obtain the sound reference information corresponding to each text; and / or, the sound reference information includes a role category and an emotion category, and based on the sound reference information, generating voice data for each text, including: determining a target timbre corresponding to the text based on the role category corresponding to the text; performing speech synthesis on the text based on the target timbre and emotion category corresponding to the text to obtain the voice data of the text.

[0108] Among them, the information recognition network is used to process the second text features of each text and the entity features of each named entity to obtain the sound reference information corresponding to each text, including: based on the entity features of each named entity, determining the entity relationship between each named entity, and based on the second text features of each text, obtaining the text relationship between each text; based on the entity relationship and the text relationship, determining the correspondence between each named entity and each text, and using the correspondence between each named entity and each text, obtaining the sound reference information corresponding to each text.

[0109] Among them, determining the screen change content of the object corresponding to each text includes: detecting the part area corresponding to the preset part of each object in the static picture book, extracting key points in the part area of ​​the object, and obtaining reference key point position information of the object, wherein the reference key point position information includes the position information of multiple key points in the preset part of the object in the static picture book; for each text, taking the object and voice data corresponding to the text as the target object and the target voice, based on the reference key point position information of the target object and the target voice, obtaining the position change of multiple key points of the target object in the process of playing the target voice, and determining the screen change content of the target object based on the corresponding position change of the target object.

[0110] Among them, the position change corresponding to the target object includes several groups of key point position offsets of the target object, different groups of key point position offsets correspond to different time segments of the target speech, and each group of key point position offsets includes position offsets of multiple key points of the target object corresponding to the time segment; based on the position change corresponding to the target object, the picture change content of the target object is determined, including: for each time segment, based on the key point position offsets of the target object corresponding to the time segment, determining the target part image of the target object corresponding to the time segment; wherein the target part image includes a preset part, and several frames of target part images of the target object corresponding to different time segments constitute the picture change content of the target object.

[0111] Wherein, based on the reference key point position information of the target object and the target voice, the position change of multiple key points of the target object in the process of playing the target voice is obtained, including: extracting the voice features of the target voice; using the position prediction network to process the reference key point position information and the voice features of the target object to obtain several groups of key point position offsets of the target object; and / or, based on the key point position offsets of the target object corresponding to the time segment, determining the target part image of the target object corresponding to the time segment, including: based on the reference key point position information of the target object and the key point position offsets of the corresponding time segment, obtaining the region information of the target object corresponding to the time segment, wherein the region information includes the position offset of at least some pixels in the part region of the target object; based on the position offsets of at least some pixels and the reference part image of the target object, obtaining the target part image of the target object corresponding to the time segment, wherein the reference part image includes a preset part of the target object.

[0112] Among them, the reference key point position information and speech features of the target object are processed by using a position prediction network to obtain several groups of key point position offsets of the target object, including: inputting the reference key point position information, speech features, and reference factors of the target object into the position prediction network to obtain several groups of key point position offsets of the target object, wherein the reference factors include noise and encoding features of the emotional category of the text corresponding to the target object, the noise represents the head movement and the facial expressions in the preset parts of the target object when speaking, and different emotional categories correspond to different motion templates of the preset parts; based on the position offsets of at least part of the pixels and the reference part image of the target object, a target part image of the target object corresponding to the time segment is obtained, including: using the position offsets of at least part of the pixels to deform the reference part image to generate a deformed image of the target object; using the deformed image, feature information of the reference part image, and the key point position offsets of the corresponding time segment to obtain the target part image of the target object corresponding to the time segment.

[0113] Among them, the area information also includes a region mask, which indicates the position of a preset area in the target part image, and the preset area is invisible in the corresponding reference part image and visible in the target part image; after obtaining the target part image of the target object corresponding to the time segment based on the position offset of at least part of the pixels and the reference part image of the target object, the method also includes: filling the preset area in the target part image with content.

[0114] Among them, the picture change content of the object includes several frames of target part images of the object with respect to a preset part, and different frames of target part images correspond to different time segments of the voice data; based on the voice data and the picture change content, the animation picture book data corresponding to the static picture book is generated, including: for each time segment, the background image of the static picture book is spliced ​​with the target part image of the corresponding time segment to obtain the picture book frame of the corresponding time segment; based on the voice data and several picture book frames corresponding to different time segments of the voice data, the animation picture book data is generated.

[0115] Among them, before splicing the background image of the static picture book with the target part image of the corresponding time segment to obtain the picture book frame of the corresponding time segment, the method also includes: obtaining the positional relationship between multiple key points of the preset part of the object and a preset number of associated background points, wherein the associated background points are located within a preset distance from the boundary of the preset part of the object in the static picture book; after splicing the background image of the static picture book with the target part image of the corresponding time segment to obtain the picture book frame of the corresponding time segment, the method also includes: adjusting the position of the associated background points in the picture book frame based on the corresponding positional relationship of the objects.

[0116] Among them, the preset part is the face; and / or, the method also includes: selecting at least one object from a static picture book as a dialogue object; based on the preset content and / or the text content in the static picture book, driving the dialogue object to communicate with the user.

[0117] See also Fig.10 , Fig.10 It is a schematic diagram of the framework of an implementation method of an animation picture book generation device provided by the present application.

[0118] The animation picture book generating device 100 includes a memory 101 and a processor 102 coupled to each other, the memory 101 stores program instructions, and the processor 102 is used to execute the program instructions to implement the steps in any of the above method embodiments. Specifically, the animation picture book generating device 100 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

[0119] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above method embodiments. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0120] See also Fig.11 , Fig.11 The computer-readable storage medium 110 stores program instructions 111, which are used to implement the steps in any of the above method embodiments when executed by a processor.

[0121] The computer-readable storage medium 110 may specifically be a medium that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the computer program. The server may send the stored computer program to other devices for execution, or may also execute the stored computer program itself.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0123] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0125] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating an animated picture book, characterized in that: include: Identify several objects and some text in a static picture book; Determine a correspondence between each of the objects and each of the texts, wherein the correspondence indicates that the text is narrated by the object corresponding to the text; Generate voice data of each of the texts, and determine the screen change content of the object corresponding to each of the texts, wherein the screen change content represents the change of the preset part of the object during the process of playing the voice data corresponding to the text; Based on the voice data and the picture change content, generating animation picture book data corresponding to the static picture book; Wherein, the generating of the voice data of each of the texts comprises: Extracting entity features of at least one named entity from the plurality of texts; Determining entity relations between the named entities based on entity features of the named entities, and obtaining text relations between the texts based on second text features of the texts; Based on the entity relationship and the text relationship, determining the correspondence between each of the named entities and each of the texts, and using the correspondence between each of the named entities and each of the texts to obtain sound reference information corresponding to each of the texts, wherein the sound reference information includes at least one of a role category and an emotion category; Based on the sound reference information, voice data of each of the texts is generated.

2. The method according to claim 1, characterized in that The identifying of a plurality of objects and a plurality of texts in a static picture book includes: Performing object detection on the static picture book to obtain a detection result including the plurality of objects; and Perform text recognition on the static picture book to obtain the plurality of texts.

3. The method according to claim 1, characterized in that: The determining of the correspondence between the objects and the texts includes: Extracting image features of each of the objects from the static picture book using a first extraction network; and For each of the texts, extracting a plurality of word vectors of the text from the text using a second extraction network, and aggregating the plurality of word vectors of the text to obtain a first text feature of the text; Using a relationship recognition network to discriminate the image features of each object and the first text features of each text, and obtain the similarity between each text and each object; Based on the similarity, the corresponding relationship between each of the objects and each of the texts is obtained.

4. The method according to claim 1, characterized in that: The generating of the voice data of each of the texts based on the sound reference information comprises: Determining a target timbre corresponding to the text based on the character category corresponding to the text; Based on the target timbre and emotion category corresponding to the text, speech synthesis is performed on the text to obtain speech data of the text.

5. The method according to claim 1, characterized in that The step of determining the screen change content of the object corresponding to each of the texts includes: Detecting a part area corresponding to a preset part of each of the objects in the static picture book, extracting key points from the part area of ​​the object, and obtaining reference key point position information of the object, wherein the reference key point position information includes position information of a plurality of key points in the preset part of the object in the static picture book; For each of the texts, the object and voice data corresponding to the text are used as the target object and target voice. Based on the reference key point position information of the target object and the target voice, the position changes of multiple key points of the target object during the playing of the target voice are obtained. Based on the position changes corresponding to the target object, the picture change content of the target object is determined.

6. The method according to claim 5, characterized in that The position change corresponding to the target object includes several groups of key point position offsets of the target object, different groups of key point position offsets correspond to different time segments of the target speech, and each group of key point position offsets includes position offsets of multiple key points of the target object corresponding to the time segment; The determining the screen change content of the target object based on the position change corresponding to the target object includes: For each of the time segments, based on the position offset of the key points of the target object corresponding to the time segment, the target part image of the target object corresponding to the time segment is determined; wherein, the target part image includes the preset part, and several frames of the target part images of the target object corresponding to different time segments constitute the picture change content of the target object.

7. The method according to claim 6, characterized in that The obtaining, based on the reference key point position information of the target object and the target voice, position changes of multiple key points of the target object during the playing of the target voice includes: Extracting speech features of the target speech; The reference key point position information of the target object and the speech features are processed using a position prediction network to obtain several groups of key point position offsets of the target object.

8. The method according to claim 6, characterized in that The determining, based on the key point position offset of the target object corresponding to the time segment, a target part image of the target object corresponding to the time segment, comprises: Based on the reference key point position information of the target object and the key point position offset corresponding to the time segment, obtaining the region information of the target object corresponding to the time segment, wherein the region information includes the position offset of at least part of the pixel points in the part region of the target object; Based on the position offset of at least part of the pixel points and the reference part image of the target object, a target part image of the target object corresponding to the time segment is obtained, wherein the reference part image includes the preset part of the target object.

9. The method according to claim 7, characterized in that: The method of using a position prediction network to process the reference key point position information of the target object and the speech features to obtain a plurality of groups of key point position offsets of the target object includes: Inputting the reference key point position information of the target object, the speech features, and the reference factors into the position prediction network to obtain several groups of key point position offsets of the target object, wherein the reference factors include noise and encoding features of the emotional category of the text corresponding to the target object, the noise represents the head movement and the expression in the preset part when the target object speaks, and different emotional categories correspond to different motion templates of the preset part; The obtaining, based on the position offset of at least part of the pixels and the reference part image of the target object, the target part image corresponding to the time segment comprises: Deforming the reference part image using the position offset of at least part of the pixels to generate a deformed image of the target object; The target part image of the target object corresponding to the time segment is obtained by using the deformed image, the feature information of the reference part image, and the key point position offset corresponding to the time segment.

10. The method according to claim 8, characterized in that The region information further includes a region mask, wherein the region mask indicates a position of a preset region in the target part image, wherein the preset region is invisible in the corresponding reference part image and visible in the target part image; After obtaining the target part image of the target object corresponding to the time segment based on the position offset of at least part of the pixels and the reference part image of the target object, the method further includes: The preset area in the target part image is filled with content.

11. The method according to claim 1, characterized in that: The image change content of the object includes a plurality of frames of target part images of the object with respect to a preset part, and different frames of the target part images correspond to different time segments of the voice data; and generating the animation picture book data corresponding to the static picture book based on the voice data and the image change content includes: For each of the time segments, splicing the background image of the static picture book with the target part image corresponding to the time segment to obtain a picture book frame corresponding to the time segment; The animation picture book data is generated based on the voice data and a plurality of picture book frames corresponding to different time segments of the voice data.

12. The method according to claim 11, characterized in that Before splicing the background image of the static picture book with the target part image corresponding to the time segment to obtain the picture book frame corresponding to the time segment, the method further includes: Acquire a positional relationship between a plurality of key points of a preset part of the object and a preset number of associated background points, wherein the associated background points are located within a preset distance from a boundary of the preset part of the object in the static picture book; After splicing the background image of the static picture book with the target part image corresponding to the time segment to obtain the picture book frame corresponding to the time segment, the method further includes: Based on the positional relationship corresponding to the object, the position of the associated background point in the picture book frame is adjusted.

13. The method according to claim 1, characterized in that The preset part is face; and / or, The method further comprises: Select at least one object from the static picture book as a dialogue object; Based on preset content and / or text content in the static picture book, the dialogue object is driven to have a dialogue with the user.

14. An animation picture book generating device, characterized in that: The device comprises: A recognition module, the recognition module is used to recognize a number of objects and a number of texts in a static picture book; A determination module, the determination module determines a correspondence between each of the objects and each of the texts, wherein the correspondence indicates that the text is narrated by the object corresponding to the text; A comprehensive module, the comprehensive module is used to generate voice data of each of the texts, and determine the screen change content of the object corresponding to each of the texts, wherein the screen change content represents the change of the preset part of the object during the process of playing the voice data corresponding to the text; A generating module, the generating module is used to generate the animation picture book data corresponding to the static picture book based on the voice data and the picture change content; Wherein, the generating of the voice data of each of the texts comprises: Extracting entity features of at least one named entity from the plurality of texts; Determining entity relations between the named entities based on entity features of the named entities, and obtaining text relations between the texts based on second text features of the texts; Based on the entity relationship and the text relationship, determining the correspondence between each of the named entities and each of the texts, and using the correspondence between each of the named entities and each of the texts to obtain sound reference information corresponding to each of the texts, wherein the sound reference information includes at least one of a role category and an emotion category; Based on the sound reference information, voice data of each of the texts is generated.

15. An animation picture book generating device, characterized in that: The device includes a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the method according to any one of claims 1 to 13.

Citation Information

Patent Citations

  • Cartoon dubbing method and apparatus based on voice synthesis

    CN106531148A

  • A story data processing method and a system for an intelligent robot

    CN109460548A

  • Voice synthesis method, device and equipment and medium

    CN111667811A

  • Text-to-speech conversion method, device, electronic equipment and storage medium

    CN112765971A

  • Animation generation method and device, storage medium and electronic equipment

    CN113538641A