Children's picture book auxiliary reading method and system based on TTS technology

By obtaining the children's voice emotions and content text constraints, building data sets and lexicons, using word vector models to analyze and convert picture book content, combined with TTS technology, the problems of personalized experience and uncommon vocabulary understanding in traditional reading methods are solved, and the problems of personalized experience and lexicon understanding in traditional reading methods are improved. Children's interest in reading and comprehension ability.

CN118711568BActive Publication Date: 2024-11-26XIAMEN SANDU EDUCATION TECH CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202411172137.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-26
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

Traditional children's picture book assisted reading methods are difficult to provide personalized experiences based on individual differences in children, and the rare and difficult vocabulary in the picture book content is difficult to effectively understand by children, resulting in a monotonous and boring reading process and affecting children's attention and interests.

Method used

By obtaining the pronunciation and emotional constraints of children and text constraints, a pronunciation and emotional data set and rare and difficult vocabulary database are constructed, and the word vector model is used for semantic analysis and transformation, adaptive content is generated, and voice-emotion conversion is combined with TTS technology to output picture book pronunciation. Ensure that the content meets the emotional needs and understanding of young children.

Benefits of technology

It realizes a personalized picture book reading experience, improves children's reading interest and comprehension ability, and promotes language and emotional development.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118711568B_ABST
    Figure CN118711568B_ABST
Patent Text Reader

Abstract

The present application relates to the field of assisted reading technology, and provides a method and system for assisted reading of picture books for young children based on TTS technology. The method includes: obtaining reading constraints, including voice emotions and content text; constructing an emotional data set, training a TTS model, generating voice conversion information and emotional features; extracting picture book content through text recognition, and generating a paging set; parsing the content to construct an uncommon word library, performing semantic conversion, obtaining adaptation data and positioning mapping; inserting conversion data, generating a combed text; performing voice conversion based on conversion information and emotional features, and outputting picture book voice. The present application solves the technical problems that traditional assisted reading methods are difficult to provide personalized experiences based on individual differences of young children, and that uncommon and difficult words in picture book content are difficult for young children to effectively understand. It achieves the effect of improving children's reading comprehension ability and promoting children's language and emotional development by combining intelligent analysis and adaptive conversion of picture book content.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of auxiliary reading technology, and in particular to a method and system for auxiliary reading of picture books for young children based on TTS technology. Background Art

[0002] With the rapid advancement of science and technology and the increasing attention paid to family education, picture book reading for children, as an important part of early childhood education, is undergoing unprecedented changes. The traditional way of reading picture books for children mainly relies on the company and explanation of parents. Although this method is warm and helpful for the establishment of parent-child relationships, it has limitations in personalized teaching, interactivity and improvement of learning efficiency. Especially in the fast-paced modern life, how to more efficiently assist children in reading picture books while maintaining their interest and concentration has become the focus of common concern of parents and educators. In this context, a method and system for assisting children in reading picture books based on TTS (Text-To-Speech) technology came into being. However, although the existing TTS technology can realize text-to-speech conversion, it often lacks the ability of emotional expression and intelligent understanding, resulting in the lack of vividness and appeal of the generated speech. At the same time, it is impossible to make real-time adjustments and optimizations according to the reading progress, interest preferences and comprehension ability of children, resulting in a monotonous and boring reading process and difficulty in maintaining the attention and interest of children. Summary of the invention

[0003] This application provides a method and system for assisting children's picture book reading based on TTS technology, aiming to solve the technical problems that traditional assistive reading methods are difficult to provide personalized experience according to individual differences of children, and that unfamiliar and difficult words in picture book content are difficult for children to effectively understand.

[0004] In view of the above problems, the present application provides a method and system for assisting reading of picture books for young children based on TTS technology.

[0005] The first aspect disclosed in the present application provides a method for assisting picture book reading for young children based on TTS technology, the method comprising: obtaining reading constraints for young children, the reading constraints for young children comprising speech emotion constraints and content text constraints; collecting data according to the speech emotion constraints, constructing a speech emotion data set, using the speech emotion data set to train a speech emotion recognition TTS model, and generating speech conversion information and emotion feature information; extracting content from picture books for young children through a text recognition model, and generating a picture book content paging set; parsing the picture book content text of the picture book content paging set according to the content text constraints, constructing a rare and difficult word library, performing semantic analysis and conversion through a word vector model, obtaining content adaptation conversion data, and establishing a positioning mapping association between the content adaptation conversion data and the picture book content paging set; inserting the content adaptation conversion data into the picture book content paging set based on the positioning mapping, and generating a picture book combing text; performing speech emotion conversion on the picture book combing text according to the speech conversion information and emotion feature information, and outputting picture book speech.

[0006] Another aspect disclosed in the present application provides a children's picture book auxiliary reading system based on TTS technology, the system comprising: a constraint acquisition module, the constraint acquisition module is used to obtain children's reading constraints, the children's reading constraints include voice emotion constraints and content text constraints; a model training module, the model training module is used to collect data according to the voice emotion constraints, construct a voice emotion data set, use the voice emotion data set to train a voice emotion recognition TTS model, and generate voice conversion information and emotion feature information; a content extraction module, the content extraction module is used to extract content from children's picture books through a text recognition model, and generate a picture book content paging set. ; an analysis and conversion module, the analysis and conversion module is used to perform picture book content text analysis on the picture book content paging set according to the content text constraints, build a rare and difficult word library, perform semantic analysis and conversion through a word vector model, obtain content adaptation conversion data, and establish an association between the content adaptation conversion data and the picture book content paging set positioning mapping; a data insertion module, the data insertion module is used to insert the content adaptation conversion data into the picture book content paging set based on positioning mapping to generate a picture book combing text; an emotion conversion module, the emotion conversion module is used to perform voice emotion conversion on the picture book combing text according to the voice conversion information and emotion feature information, and output a picture book voice.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] The above-mentioned method for assisting reading of picture books for young children based on TTS technology first clarifies the speech emotion constraints and content text constraints of children's reading. These constraints help ensure that the output picture book reading content meets the emotional needs of young children. In view of the speech emotion constraints, a large amount of speech emotion data is collected, which contains speech samples under different emotional states. By training a speech emotion recognition TTS model, these text contents are converted into speech conversion information and emotional feature information. Subsequently, all text contents are extracted from the picture book using the text recognition model and sorted into pagination sets. According to the content text constraints, these text contents are parsed to identify the uncommon words and difficult contents therein. In order to help young children better understand these difficulties, an uncommon and difficult word library is established, and these words are semantically analyzed and converted through the word vector model to generate content adaptation conversion data that is easier for young children to understand. At the same time, a positioning mapping relationship between these conversion data and the pagination set of picture book content is established to ensure that the converted content can be accurately inserted into the corresponding position when reading aloud. Afterwards, based on the voice conversion information and emotional feature information, the picture book text is sorted out for voice emotion conversion, and the picture book voice is output to accurately convey the content of the story, and attract the children's attention through rich emotional expression, thereby enhancing their reading interest and comprehension ability.

[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 The figure is a flowchart of a method for assisting picture book reading for young children based on TTS technology in one embodiment.

[0012] Figure 2 This is an architecture diagram of a children's picture book auxiliary reading system based on TTS technology in one embodiment.

[0013] Explanation of the accompanying drawings: constraint acquisition module 1, model training module 2, content extraction module 3, analysis and conversion module 4, data insertion module 5, emotion conversion module 6. DETAILED DESCRIPTION

[0014] The embodiments of the present application provide a method and system for assisting reading of picture books for children based on TTS technology, thereby solving the technical problems that traditional assistive reading methods are difficult to provide a personalized experience based on individual differences of children, and that unfamiliar and difficult words in the content of picture books are difficult for children to effectively understand.

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0016] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.

[0017] Embodiment 1, as Figure 1 As shown, the present application provides a method for assisting reading of picture books for young children based on TTS technology, the method comprising:

[0018] Obtaining reading constraints for young children, wherein the reading constraints for young children include voice emotion constraints and content text constraints.

[0019] In an embodiment of the present application, during the process of children's reading, in order to ensure that the reading experience not only meets the cognitive and emotional needs of children, but also promotes the language and learning development of children, the system terminal obtains preset children's reading constraints, which include voice emotion constraints and content text constraints. Among them, the voice emotion constraints mainly focus on the voice types and emotional expression characteristics preferred by children when reading or listening to stories, such as the voices of parents and grandparents, the voices of cartoon characters, etc. The content text constraints focus on screening and evaluating the content and vocabulary in children's picture books according to the age of the children to ensure their suitability and comprehensibility, and avoid overly complex or profound vocabulary and sentences. For example, for younger children, stories with rich pictures, simple plots, and short and easy-to-understand language should be selected; for older children, the complexity and depth of the story can be appropriately increased.

[0020] Furthermore, the present application provides the obtaining of the children's reading constraint conditions, including:

[0021] Through the user input interface, input the child's favorite characters and the child's age, the child's favorite characters include relative identities, cartoon identities, human voice identities, and other supplementary identities; according to the child's favorite characters, obtain the voice emotion constraints, the voice emotion constraints include emotion category labels and voice character identity labels; according to the child's age, perform vocabulary analysis to determine the content text constraints, the content text constraints are words that are beyond the child's age understanding range or non-life expressions.

[0022] Preferably, the system terminal receives key information about the child input from the user through the user input interface, including the child's favorite characters and the child's age. Among them, the child's favorite characters include relatives (such as parents, grandparents), cartoon characters, specific human voices (such as a host or storyteller), and any other identities that the child particularly likes. Based on this information, the system terminal analyzes the characters that the child likes and sets the corresponding application scenario labels to generate corresponding voice emotion constraints. This voice emotion constraint includes two parts: one is the emotion category label, that is, the type of emotional expression that the child may prefer when listening to a story or reading, such as gentleness, excitement, curiosity, etc.; the other is the voice character identity label, that is, the type of voice or character identity that the child particularly likes, which will guide how to imitate or adopt the voice and emotions of these specific identities when selecting or generating voice content. Subsequently, the system terminal performs vocabulary analysis based on the input child's age to determine the content text constraint. This step is to ensure that the vocabulary and expressions in the reading material are understandable and acceptable to the child. Content text constraints focus on two aspects: first, avoid using words that are beyond the current age range of children, which may be too abstract, complex or professional, and are not conducive to children's understanding; second, reduce the use of non-life expressions, that is, expressions that are not often used in daily life or that children are not familiar with, to maintain the closeness and authenticity of the reading content. In summary, after receiving children's favorite characters and age information through the user input interface, voice emotion constraints and content text constraints can be intelligently generated to provide a reading experience that is more in line with children's interests and cognitive levels.

[0023] Further, the present application provides obtaining the speech emotion constraint, further comprising:

[0024] An application scenario label of a favorite character identity is set, wherein the application scenario label includes a time period label and an emotion type label; voice features and emotion type features are traversed and matched according to the time period label and the emotion type label to generate the voice emotion constraint.

[0025] Optionally, in order to obtain the voice emotion constraints of children's reading, the system terminal first obtains the children's activity habits and emotional needs in different time periods. This is determined based on the children's daily life information. Through this information, multiple typical application scenarios can be defined, such as bedtime story time, morning wake-up time, lunch break time, parent-child game time, etc. For each defined application scenario, the system terminal assigns one or more time period tags to it. The time period tag clearly indicates the time period when the scene usually occurs, such as falling asleep at night, getting up in the morning, and lunch break. These tags help to identify the current time and select appropriate voice content and emotional expression accordingly. Subsequently, according to the characteristics and purpose of each application scenario, the emotional type tag that matches it is determined. These tags can accurately reflect the emotional atmosphere required by the child in the scenario, such as warm comfort (for bedtime stories), lively guidance (for getting up in the morning and parent-child games), and relaxed and pleasant (for lunch break or daily companionship). Afterwards, the system terminal summarizes the determined time period tags and emotional type tags to form application scenario tags, and associates the character identity preferred by the child with the above application scenario tags. This is based on the child's preference for specific character identities and the roles and emotional expressions that these characters may play in different scenarios. For example, a child may prefer to hear his mother's gentle voice telling stories before going to bed, and like to play interactive games with lively cartoon characters in the morning. Then, the system terminal traverses the pre-built voice feature library and emotional type feature library according to the determined time period label and emotional type label, and traverses the voice features and emotional type features corresponding to these two labels. Among them, the voice feature library and the emotional type feature library store label-voice features and label-emotion type features respectively. Finally, based on the results of the traversal matching, the system terminal generates a set of specific voice emotion constraints. These constraints will specify in detail what voice features and emotional type features the voice content should have under a specific time period and emotional type. For example, under the nighttime sleep time period and the soothing emotional type, voice emotion constraints including timbre, speech speed, volume, intonation, emotion and other attributes are generated.

[0026] Data is collected according to the speech emotion constraints to construct a speech emotion data set, and the speech emotion data set is used to train a speech emotion recognition TTS model to generate speech conversion information and emotion feature information.

[0027] In one embodiment, the system terminal collects speech samples that meet the constraints from multiple data sources according to the set speech emotion constraints, and constructs a speech emotion data set. The multiple data sources include recordings of daily interactions between young children, cartoon animations, etc. This speech emotion data set contains multiple emotion sample sets, which are divided based on character identity categories. Subsequently, the system terminal uses the speech emotion data set to train a recurrent neural network to obtain a speech emotion recognition model, and merges the speech emotion recognition model with the TTS model to form a speech emotion recognition TTS model. By merging the two models, the TTS model can take emotional information into account when generating speech. The trained speech emotion recognition TTS model can receive input speech and identify speech features and emotion category features. Afterwards, the system terminal generates speech conversion information and emotion features based on the speech feature recognition results and the emotion category feature recognition results. In addition, the system terminal will further optimize the model parameters by continuously collecting user feedback, thereby continuously improving the accuracy of speech emotion recognition.

[0028] Furthermore, the present application provides data collection according to the speech emotion constraint, constructing a speech emotion data set, using the speech emotion data set to train a speech emotion recognition TTS model, and generating speech conversion information and emotion feature information, including:

[0029] According to the favorite characters of the children, the character voice data is searched, and the voice data is collected and intercepted through an automatic script to build a favorite voice library; the favorite voice library is classified according to the character identity category according to the favorite voice library to build multiple identity sub-libraries; based on the multiple identity sub-libraries, the voice features and emotion categories are marked to build a multi-emotion sample set.

[0030] Optionally, the system terminal conducts a targeted voice data search based on the children's favorite characters. During the search process, the automatic script technology is used to collect the voice data of these characters from data sources such as children's daily interaction recordings and cartoon animations. Subsequently, the collected voice data is intercepted and sorted, and the redundant parts are removed to ensure that each voice segment is clear, complete and meets the requirements. All the sorted voice data are then aggregated into a favorite voice library, which contains voice samples of various characters that children like. After that, the favorite voice library is classified according to the identity category of the characters. For example, the voice data of cartoon characters is classified into one sub-library, and the voice data of family members is classified into another sub-library. Through classification processing, multiple identity sub-libraries are constructed, each of which contains voice samples of a specific identity category. This classification method helps to make subsequent emotion labeling and model training more efficient and accurate. Then, based on the multiple identity sub-libraries that have been classified, voice features and emotion labeling are performed. Analyze the speech features (such as timbre, speaking speed, intonation, etc.) and the emotions expressed (such as happiness, sadness, anger, comfort, etc.) of each speech sample, and emotionally tag each speech sample to ensure that each sample corresponds to a clear emotional label. Finally, all the labeled speech samples are aggregated into a multi-emotion sample set. This sample set not only contains rich speech data, but also covers a variety of different emotion types, providing a solid foundation for the subsequent training of speech emotion recognition models. The speech emotion recognition model is obtained by training the recurrent neural network with the multi-emotion sample set; the speech emotion recognition model is integrated into the TTS model to identify speech features and emotion category features, and the speech conversion information is generated based on the identified speech features, and the emotion feature information is obtained by injecting emotion into the speech conversion information based on the emotion category features.

[0031] Optionally, after obtaining the multi-emotion sample set, the system terminal designs the network structure of the recurrent neural network (RNN) according to the data characteristics, including the number of neurons in the input layer, hidden layer and output layer, activation function, etc. Subsequently, the multi-emotion sample set is divided into a training set, a validation set and a test set, and then the weights and bias parameters of the RNN are randomly initialized. After that, the RNN is iteratively trained using the training set data. In each iteration, the speech sample is input into the RNN, the loss between the output and the true emotion label is calculated using the cross entropy loss, and the model parameters are updated by the back propagation algorithm. During the training process, the validation set is used regularly to evaluate the performance of the RNN. If the performance of the RNN on the validation set begins to decline, that is, overfitting occurs, the training is stopped in advance, and according to the performance evaluation results of the validation set, the network structure, learning rate and other parameters of the RNN are adjusted to obtain better performance. After the training is completed, the trained RNN is evaluated using the test set to verify its performance in practical applications. After the verification is passed, the current RNN is output as the final speech emotion recognition model.

[0032] The system terminal uses the deep learning framework TensorFlow to build the Tacotron 2 model, including an encoder (processing text input), an attention mechanism (aligning text and speech frames), a decoder (generating a mel spectrogram), and a vocoder (converting a mel spectrogram into a waveform). Subsequently, the Tacotron 2 model is trained using data containing text and corresponding speech, and the training process is similar to the aforementioned training of the speech emotion recognition model. After training and testing, the system terminal outputs the current Tacotron 2 model as a TTS model. Afterwards, the system terminal integrates the trained speech emotion recognition model into the TTS model so that the two models can collaborate to identify speech features and emotion category features. During the speech generation process of the TTS model, the system terminal extracts recognition speech features, such as mel spectrograms. The speech generated by the TTS model is then input into the speech emotion recognition model to identify the emotion category features therein. Then, based on the emotion recognition results, the system terminal designs an emotion injection strategy, which involves adjusting parameters such as the volume, speech rate, and pitch of the speech to meet specific emotional expressions. Then, according to this emotion injection strategy, the recognized speech features are adjusted to generate emotion feature information to reflect the corresponding emotional color.

[0033] The text recognition model is used to extract the content of children's picture books and generate a page set of picture book content.

[0034] In one embodiment, the system terminal uses the same training process as above to train the text recognition model based on the sample picture book image set and the sample text content. The text recognition model includes a convolutional layer (CNN), a recurrent layer (RNN), and a transcription layer (CTC). After constructing the text recognition model, the system terminal uses the trained text recognition model to perform character recognition on the image of the children's picture book. This model can recognize the text in the image and convert it into a computer-readable text format. After obtaining the text content of the picture book, the system terminal organizes the recognized text in page order according to the narrative structure and content logic of the picture book to form a picture book content paging set. This paging set contains the content of each page of the picture book, as well as possible relationship information between pages, which facilitates further processing of subsequent steps such as text parsing and semantic analysis.

[0035] The picture book content page set is parsed according to the content text constraints, a rare and difficult word library is constructed, semantic analysis and conversion are performed through a word vector model, content adaptation conversion data is obtained, and a positioning mapping association between the content adaptation conversion data and the picture book content page set is established.

[0036] In one embodiment, the system terminal uses natural language processing technology to perform text analysis on the pagination set of the picture book content, extracts the text content page by page, and performs word segmentation processing on the text content to divide the vocabulary list. Subsequently, the vocabulary list is identified according to the determined content text constraints, and the rare or difficult words appearing in the text are marked, which may be difficult for young children to understand during the reading process. After obtaining the rare or difficult words, the system terminal constructs a rare and difficult word library based on the analysis results. This word library contains all the words in the picture book that need special explanation. Each word is accompanied by its original form and its specific usage in the context of the picture book. Afterwards, the system terminal uses a pre-built word vector model to perform semantic analysis and conversion on the words in the rare and difficult word library. The word vector model can capture the semantic relationship between words and help understand the deep meaning and usage of words. Then, for each rare or difficult word in the word library, the word vector model can generate corresponding colloquial interpretation or conversion data, that is, content adaptation conversion data. These content adaptation conversion data are presented in a language and expression that is easy for young children to understand, aiming to help young children better understand the meaning and usage of words. Finally, the system terminal establishes a positioning mapping association between the content adaptation conversion data and the picture book content paging set. This association ensures that when reading the picture book, the conversion data related to the current page content can be quickly found and displayed, thereby improving the reading experience and comprehension effect.

[0037] Further, the present application provides the method of performing semantic analysis and conversion through a word vector model to obtain content-adaptive conversion data, which includes:

[0038] Collect text corpus data and build a corpus, which includes multiple types of texts; use natural language processing technology to segment the texts in the corpus and divide them into vocabulary lists; use the Word2Vec algorithm to initialize the model and set training parameters, including vector dimension, window size, and minimum word frequency.

[0039] Preferably, in order to construct a word vector model, the system terminal first collects and organizes multiple types of text data, including but not limited to daily conversations, nursery rhymes, children's stories, fairy tales, and news. These diverse text types help to build a rich and comprehensive corpus, providing a solid data foundation for subsequent natural language processing and model training tasks. Subsequently, the text in the corpus is preprocessed using natural language processing technology, including removing non-text content such as punctuation, numbers, special characters, and converting the text into a unified encoding format, such as UTF-8. Afterwards, the text is segmented using an existing statistical language model, that is, the continuous text is divided into a vocabulary list, which is an important prerequisite for building a word vector. The system terminal uses the Word2Vec algorithm as a word vector generation tool to initialize the word vector model. The Word2Vec algorithm maps vocabulary to vectors in a high-dimensional space through training, so that semantically similar vocabulary is close in distance in the vector space. During the initialization process, the system terminal sets training parameters, including vector dimension, window size, and minimum word frequency. Among them, the vector dimension determines the length of the word vector, which is an important factor affecting the expressive power of the model. A value between 100 and 300 is usually chosen, and the specific value needs to be adjusted according to task requirements and computing resources. The window size defines the range of context words considered when predicting the target word. It is common to set it between 5 and 10. Smaller windows can capture closer semantic relationships, while larger windows may introduce more noise. The minimum word frequency is used to filter out low-frequency words to reduce the complexity of the model and noise interference. A common value is 5, which means that only words that appear at least 5 times in the corpus are retained.

[0040] The word vector model is obtained by using a word segmentation corpus to learn the word context relationship and embed the words into a high-dimensional vector space. The word vector is gradually adjusted by optimizing the objective function so that semantically similar words are close to each other in the vector space until a close distance threshold is reached.

[0041] Preferably, after the word vector model is initialized, the system terminal uses the corpus of word segmentation to train the word vector model. The word vector model can capture the semantic similarity between words by using the word context relationship in the corpus, that is, other words that appear around a word in the text. During the training process, the system terminal initializes a high-dimensional vector space and assigns an initial vector representation to each word in the corpus. Subsequently, the word vector model traverses the text in the corpus and gradually adjusts the values ​​of these vectors according to the contextual relationship of the words. The adjustment process is achieved by optimizing the objective function, which measures whether the distribution of word vectors in the vector space is reasonable, that is, the distance between semantically similar words in the vector space is as close as possible, while semantically dissimilar words should be kept at a farther distance. In order to achieve this goal, the system terminal will continuously iterate and update the value of the word vector until a stable state is reached, that is, the position of the word vector in the vector space no longer changes significantly. During the training process, the system terminal will set a close distance threshold to determine whether two word vectors are close enough to be considered to have similar semantics. When the training process is completed, the system terminal obtains the final word vector model, which maps each word in the corpus to a high-dimensional vector space.

[0042] Further, the present application provides the method of using the word segmentation processed corpus for model training, learning the vocabulary context relationship, and embedding the vocabulary into a high-dimensional vector space, gradually adjusting the word vector by optimizing the objective function so that semantically similar words are close to each other in the vector space until a close distance threshold is reached, and obtaining the word vector model, including:

[0043] The goal is to predict the context word based on a given word and set the objective function to maximize the goal: ; Use the softmax function to fit the contextual conditional probability of the current given word: ,in For the current given word The word vector of Context words The word vector is W, and W is the size of the vocabulary; the word vector is adjusted based on the conditional probability, and the objective function is optimized so that semantically similar words are close to each other in the vector space until a distance threshold is reached, thereby obtaining the word vector model.

[0044] Optionally, during the training process of the word vector model, the system terminal predicts the context word of a given word as a goal, and sets an objective function to maximize the goal. The specific objective function is as follows: , where T is the total number of words in the corpus, indicating the size of the vocabulary set. For the current given word, is the vocabulary whose context word is to be predicted. is the context word, which appears in the given word The words j positions before and after the given word. c is the size of the context window, which indicates how many words are considered as context before and after the given word. t is the position of the current given word text, that is, the index in the entire corpus. j is the position relative to the current given word in the context window, indicating the distance from the current word. Subsequently, the softmax function is used to calculate the context conditional probability of the given word, and the value of the word vector is adjusted based on this conditional probability. The softmax function is as follows: ;in, For the current given word The word vector is the representation of the word in the vector space, which is used to capture the semantic information of the word. Context words The word vector of It is similar to , but it represents the vector representation of the context word. W is the size of the vocabulary, that is, the number of different words in the corpus. After that, the value of the word vector is adjusted based on the calculated conditional probability. By optimizing the objective function, semantically similar words will be close to each other in the vector space until the distance between them reaches a certain threshold. In this way, the system terminal obtains a word vector model that can reflect the semantic relationship between words.

[0045] Furthermore, the present application provides the method of performing semantic analysis and conversion through a word vector model to obtain content adaptation conversion data, including:

[0046] Obtain conversion vocabulary of the word vector model; identify and screen the conversion vocabulary based on the content text constraint to determine matching conversion vocabulary; when matching conversion vocabulary exists, generate the content adaptation conversion data based on the matching conversion vocabulary; when matching conversion vocabulary does not exist, screen matching conversion vocabulary based on the constituent characters of the vocabulary to obtain the spoken interpretation content of the matching conversion vocabulary; use preset spoken interpretation conjunctions to splice with the spoken interpretation content, and generate the content adaptation conversion data in the preset conversion format of the matching conversion vocabulary+spoken interpretation conjunctions+spoken interpretation content.

[0047] Optionally, the system terminal extracts the required vocabulary from the trained word vector model. These vocabulary are learned by the model during the training process, and each vocabulary corresponds to a high-dimensional vector that can reflect the semantic relationship between the vocabulary. Subsequently, these extracted vocabulary are identified and screened according to the given content text constraints. Through this process, the system terminal can determine which vocabulary is the matching conversion vocabulary, that is, those vocabulary that both meets the content constraints and is suitable for conversion. If matching conversion vocabulary is found, the system terminal directly generates content adaptation conversion data based on these vocabulary. This conversion data is a restatement, simplification, translation or any other form of conversion of the original content, depending on the actual needs and purposes. However, if no matching conversion vocabulary is found, the system terminal will screen matching conversion vocabulary based on the constituent characters of the vocabulary. This is because some vocabulary may not appear directly in the word vector model, but their constituent characters may be familiar and common. In this way, some indirect but still related vocabulary can be found for conversion. Once such matching conversion vocabulary is found, the system terminal will further obtain the spoken interpretation content of these vocabulary. This is because spoken interpretation is usually simpler and easier to understand, suitable for informal occasions or scenarios that require quick understanding. Afterwards, the spoken explanation contents are spliced ​​with the matching conversion words using the preset spoken explanation connectors to form a complete content adaptation conversion data that conforms to the preset conversion format. This conversion format is usually fixed, for example, matching conversion words + spoken explanation connectors + spoken explanation contents, in order to maintain the consistency and readability of the conversion data.

[0048] The content adaptation conversion data is inserted into the picture book content paging set based on the positioning mapping to generate a picture book combing text.

[0049] In one embodiment, the system terminal uses the set positioning mapping to match the content adaptation conversion data to the corresponding position of the picture book content paging. Once the insertion position is determined, the system terminal inserts the content adaptation conversion data one by one into the corresponding position of the picture book content paging set. In this process, it may be necessary to adjust the size, font, color and other attributes of the text to ensure that they are consistent with the rest of the picture book and easy for readers to read and understand. Through this process, a picture book combing text is generated. This text not only contains the original content of the picture book, but also incorporates the new content that has been converted and adapted, making the story of the picture book more popular. At the same time, since the new content is converted according to the overall style and theme of the picture book, they can be naturally integrated into the picture book without causing abrupt or uncomfortable feelings.

[0050] The picture book text is subjected to speech emotion conversion according to the speech conversion information and the emotion feature information, and the picture book speech is output.

[0051] In one embodiment, the system terminal uses voice conversion information and emotional feature information to perform voice emotion conversion on the picture book combing text. In this process, the system terminal adjusts the tone, volume, speaking speed and other parameters of the voice according to the text content, emotional features and voice conversion rules to accurately convey the emotional information in the text. For example, when describing a happy scene, the voice will sound brighter and more cheerful; when describing a sad scene, it may become low and slow. Finally, the text that has undergone voice emotion conversion is converted into picture book voice and output.

[0052] In summary, the embodiments of the present application have at least the following technical effects:

[0053] The embodiment of the present application obtains the reading constraints of the children, including voice emotion and content text constraints, and performs data collection and model training according to these constraints. Through the construction of the voice emotion data set and the training of the TTS model, the voice emotion conversion of the picture book content is realized. At the same time, the text recognition model is used to extract the picture book content, and through semantic analysis and conversion, it is ensured that the content meets the understanding level of the children. The method performs semantic analysis through the word vector model, generates adaptive conversion data, and finally associates these data with the picture book content paging set positioning mapping to generate a picture book combing text that meets the voice emotion constraints. The overall process integrates voice data collection, emotion labeling, the construction of multi-emotion sample sets, and the application of recurrent neural networks, so that children can read in a personalized voice emotion environment and improve their reading comprehension ability. These technical effects jointly solve the technical problems that traditional auxiliary reading methods are difficult to provide personalized experiences according to individual differences of children, and that the rare and difficult words in the picture book content are difficult to be effectively understood by children, and realize the effect of improving children's reading comprehension ability and promoting children's language and emotional development by combining intelligent analysis and adaptive conversion of picture book content.

[0054] Embodiment 2 is based on the same inventive concept as the method for assisting children's picture book reading based on TTS technology in the above embodiment. Figure 2 As shown, the present application provides a picture book auxiliary reading system for children based on TTS technology, the system comprising: a constraint acquisition module 1: the constraint acquisition module 1 is used to obtain the reading constraint conditions for children, the reading constraint conditions for children include voice emotion constraint and content text constraint;

[0055] Model training module 2: the model training module 2 is used to collect data according to the speech emotion constraint, construct a speech emotion data set, use the speech emotion data set to train the speech emotion recognition TTS model, and generate speech conversion information and emotion feature information;

[0056] Content extraction module 3: the content extraction module 3 is used to extract the content of the children's picture book through a text recognition model to generate a picture book content page set;

[0057] Analysis and conversion module 4: the analysis and conversion module 4 is used to perform picture book content text analysis on the picture book content page set according to the content text constraint, build a rare and difficult word library, perform semantic analysis and conversion through a word vector model, obtain content adaptation conversion data, and establish a positioning mapping association between the content adaptation conversion data and the picture book content page set;

[0058] Data insertion module 5: the data insertion module 5 is used to insert the content adaptation conversion data into the picture book content paging set based on positioning mapping to generate a picture book combing text;

[0059] Emotion conversion module 6: The emotion conversion module 6 is used to perform speech emotion conversion on the picture book combing text according to the speech conversion information and emotion feature information, and output the picture book speech.

[0060] Furthermore, the constraint acquisition module 1 is also used to execute the following method:

[0061] Through the user input interface, input the child's favorite characters and the child's age, the child's favorite characters include relative identities, cartoon identities, human voice identities, and other supplementary identities; according to the child's favorite characters, obtain the voice emotion constraints, the voice emotion constraints include emotion category labels and voice character identity labels; according to the child's age, perform vocabulary analysis to determine the content text constraints, the content text constraints are words that are beyond the child's age understanding range or non-life expressions.

[0062] Furthermore, the constraint acquisition module 1 is also used to execute the following method:

[0063] An application scenario label of a favorite character identity is set, wherein the application scenario label includes a time period label and an emotion type label; voice features and emotion type features are traversed and matched according to the time period label and the emotion type label to generate the voice emotion constraint.

[0064] Furthermore, the model training module 2 is also used to perform the following method:

[0065] According to the favorite characters of the children, the character's voice data is searched, and the voice data is collected and intercepted through an automatic script to build a favorite voice library; the favorite voice library is classified according to the character identity category based on the favorite voice library to build a plurality of identity sub-libraries; based on the plurality of identity sub-libraries, the voice features and emotion categories are marked to build a multi-emotion sample set; a recurrent neural network is trained through the multi-emotion sample set to obtain a voice emotion recognition model; the voice emotion recognition model is integrated into the TTS model to perform voice feature and emotion category feature recognition, and the voice conversion information is generated according to the recognized voice features, and the emotion is injected into the voice conversion information according to the emotion category features to obtain the emotion feature information.

[0066] Furthermore, the analysis and conversion module 4 is also used to perform the following method:

[0067] The invention collects text corpus data and constructs a corpus, wherein the corpus includes multiple types of texts; uses natural language processing technology to segment the texts in the corpus and divide them into a vocabulary list; uses the Word2Vec algorithm to initialize the model and set training parameters, including vector dimension, window size, and minimum word frequency; uses the segmented corpus to train the model, learns the vocabulary context relationship, and embeds the vocabulary into a high-dimensional vector space, and gradually adjusts the word vector by optimizing the objective function so that semantically similar words are close to each other in the vector space until a close distance threshold is reached, thereby obtaining the word vector model.

[0068] Furthermore, the analysis and conversion module 4 is also used to perform the following method:

[0069] The goal is to predict the context word based on a given word and set the objective function to maximize the goal: , where T is the total number of words in the corpus, For the current given word, is the context word, c is the context window size, t is the position of the current given word text, and j is the position relative to the current given word in the context window; the softmax function is used to fit the context conditional probability of the current given word: ,in For the current given word The word vector of Context words The word vector is W, and W is the size of the vocabulary; the word vector is adjusted based on the conditional probability, and the objective function is optimized so that semantically similar words are close to each other in the vector space until a distance threshold is reached, thereby obtaining the word vector model.

[0070] Furthermore, the analysis and conversion module 4 is also used to perform the following method:

[0071] Obtain conversion vocabulary of the word vector model; identify and screen the conversion vocabulary based on the content text constraint to determine matching conversion vocabulary; when matching conversion vocabulary exists, generate the content adaptation conversion data based on the matching conversion vocabulary; when matching conversion vocabulary does not exist, screen matching conversion vocabulary based on the constituent characters of the vocabulary to obtain the spoken interpretation content of the matching conversion vocabulary; use preset spoken interpretation conjunctions to splice with the spoken interpretation content, and generate the content adaptation conversion data in the preset conversion format of the matching conversion vocabulary+spoken interpretation conjunctions+spoken interpretation content.

[0072] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order and continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0073] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0074] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A method for assisting children's picture book reading based on TTS technology, characterized in that: The method for assisting reading of picture books for young children based on TTS technology comprises: Obtaining reading constraints for children, wherein the reading constraints for children include voice emotion constraints and content text constraints; Collect data according to the speech emotion constraints, construct a speech emotion data set, use the speech emotion data set to train a speech emotion recognition TTS model, and generate speech conversion information and emotion feature information; The text recognition model is used to extract the content of children's picture books and generate a set of picture book content pages; According to the content text constraint, the picture book content paging set is parsed, a rare and difficult word library is constructed, semantic analysis and conversion are performed through a word vector model to obtain content adaptation conversion data, and a positioning mapping association between the content adaptation conversion data and the picture book content paging set is established; Inserting the content adaptation conversion data into the picture book content paging set based on the positioning mapping to generate a picture book combing text; Performing voice emotion conversion on the picture book combing text according to the voice conversion information and the emotion feature information, and outputting the picture book voice; The obtaining of the reading constraint conditions for young children includes: Input the child's favorite characters and the child's age through the user input interface, wherein the child's favorite characters include relatives, cartoons, human voices, and other supplementary identities; According to the child's favorite character, the voice emotion constraint is obtained, wherein the voice emotion constraint includes an emotion category label and a voice character identity label; Performing vocabulary analysis according to the age of the child to determine the content text constraints, wherein the content text constraints are words that are beyond the comprehension range of the child or are not life-like expressions; Obtaining the speech emotion constraint further includes: Setting application scenario tags for favorite character identities, wherein the application scenario tags include time period tags and emotion type tags; According to the time period label and the emotion type label, voice features and emotion type features are traversed and matched to generate the voice emotion constraint; Data collection is performed according to the speech emotion constraint, a speech emotion data set is constructed, and the speech emotion recognition TTS model training is performed using the speech emotion data set to generate speech conversion information and emotion feature information, including: According to the children's favorite characters, searching for character voice data, collecting and intercepting voice data through automatic scripts, and building a favorite voice library; Classifying the preferred voice library according to the character identity category according to the preferred voice library to construct multiple identity sub-libraries; Marking speech features and emotion categories based on the multiple identity sub-libraries to construct a multi-emotion sample set; Performing recurrent neural network training on the multi-emotion sample set to obtain a speech emotion recognition model; The speech emotion recognition model is integrated into the TTS model to perform speech feature and emotion category feature recognition, and the speech conversion information is generated according to the recognized speech features. The emotion is injected into the speech conversion information according to the emotion category features to obtain the emotion feature information.

2. The method for assisting children's picture book reading based on TTS technology as claimed in claim 1, characterized in that: The semantic analysis and conversion are performed through the word vector model to obtain content adaptation conversion data, which previously includes: Collecting text corpus data and building a corpus, wherein the corpus includes multiple types of texts; Using natural language processing technology, segmenting the text in the corpus into a vocabulary list; Use the Word2Vec algorithm to initialize the model and set the training parameters, including vector dimension, window size, and minimum word frequency; The word vector model is obtained by using a word segmentation corpus to learn the word context relationship and embed the words into a high-dimensional vector space. The word vector is gradually adjusted by optimizing the objective function so that semantically similar words are close to each other in the vector space until a close distance threshold is reached.

3. The method for assisting children's picture book reading based on TTS technology as claimed in claim 2, characterized in that: The word vector model is obtained by using the word segmentation processed corpus to train the model, learn the vocabulary context relationship, embed the vocabulary into a high-dimensional vector space, and gradually adjust the word vector by optimizing the objective function so that semantically similar words are close to each other in the vector space until a close distance threshold is reached, and the word vector model is obtained, including: The goal is to predict the context word based on a given word and set the objective function to maximize the goal: , where T is the total number of words in the corpus, For the current given word, is the context word, c is the context window size, t is the position of the current given word text, and j is the position of the current given word in the context window relative to the current given word; Use the softmax function to fit the contextual conditional probability of the current given word: ,in For the current given word The word vector of For context words word vector, W is the size of the vocabulary; The word vector is adjusted based on the conditional probability, and the objective function is optimized so that semantically similar words are close to each other in the vector space until a distance threshold is reached, thereby obtaining the word vector model.

4. The method for assisting children's picture book reading based on TTS technology as claimed in claim 3, characterized in that: The semantic analysis and conversion are performed through the word vector model to obtain content adaptation conversion data, including: Obtaining a conversion vocabulary of the word vector model; Identify and filter the conversion words based on the content text constraint to determine matching conversion words; When there is a matching conversion word, generating the content adaptation conversion data based on the matching conversion word; When there is no matching conversion vocabulary, the matching conversion vocabulary is screened based on the constituent characters of the vocabulary, and the spoken interpretation content of the matching conversion vocabulary is obtained; The content adaptation conversion data is generated by splicing the preset spoken interpretation connecting words with the spoken interpretation content in the preset conversion format of the matching conversion vocabulary+spoken interpretation connecting words+spoken interpretation content.

5. The children's picture book auxiliary reading system based on TTS technology is characterized by: The method for assisting reading of picture books for young children based on TTS technology according to any one of claims 1 to 4 specifically comprises: Constraint acquisition module: obtains reading constraints for children, wherein the reading constraints for children include voice emotion constraints and content text constraints; Model training module: collect data according to the speech emotion constraints, construct a speech emotion data set, use the speech emotion data set to train the speech emotion recognition TTS model, and generate speech conversion information and emotion feature information; Content extraction module: extracts content from children's picture books through a text recognition model and generates a page set of picture book content; Analysis and conversion module: performing picture book content text analysis on the picture book content page set according to the content text constraint, building a rare and difficult word library, performing semantic analysis and conversion through a word vector model, obtaining content adaptation conversion data, and establishing a positioning mapping association between the content adaptation conversion data and the picture book content page set; Data insertion module: inserting the content adaptation conversion data into the picture book content paging set based on positioning mapping to generate picture book combing text; Emotion conversion module: performs speech emotion conversion on the picture book combing text according to the speech conversion information and emotion feature information, and outputs the picture book speech.

Citation Information

Patent Citations

  • A picture book reading interaction method and system for an intelligent robot

    CN109710748A

  • Man-machine interaction method and system for picture book reading robot

    CN109940627A

  • Text conversion model training method and device, text conversion method and device and electronic equipment

    CN114706942A

  • Synonym recommendation method and device, electronic equipment and storage medium

    CN116415575A

  • Picture book reading record-based children reading level prediction method and system

    CN117217624A