Control method and device of intelligent toy, electronic equipment and storage medium

Abstract

The application provides a control method of an intelligent toy, which comprises the following steps: when the intelligent toy is in a question and answer state, acquiring voice data and / or image data of a user; performing emotion recognition processing on the voice data and / or image data by using a preset emotion recognition model to determine an emotion recognition result of the user; performing question and answer processing on question and answer data of the user based on the emotion recognition result, combining long-term memory knowledge and a target knowledge base to obtain answer data, the level of the target knowledge base corresponding to a current growth value of the intelligent toy, and the target knowledge base of different levels having knowledge data corresponding to different user ages; and controlling the intelligent toy to output the answer data. The application provides interactive content more in line with the emotion of the user by considering the emotion of the user, thereby enhancing the interactive experience of the user.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to a control method, device, electronic device, and storage medium for an intelligent toy. Background Technology

[0002] Current smart toys offer children storytelling abilities and simple voice command interaction, while also imparting knowledge during these interactions. However, because these smart toys rely on simple voice command interaction and are based on a fixed database, they output fixed content corresponding to the voice commands. Their functionality is relatively limited. Since children's emotions are unpredictable, smart toys cannot proactively understand their needs and therefore cannot provide responses that match their current mood, resulting in a poor companionship experience. Therefore, existing smart toys suffer from the problem of limited voice interaction methods, an inability to respond appropriately to the child's emotional state, and a poor companionship experience. Summary of the Invention

[0003] This invention provides a control method for smart toys, aiming to solve the problems of existing smart toys having limited interaction methods, failing to respond appropriately to a child's current emotional state, and resulting in a poor companionship experience. When the smart toy is in a question-and-answer mode, it acquires the user's voice data and / or image data, processes the voice data and / or image data using a preset emotion recognition model to determine the user's emotion recognition result, and, based on the emotion recognition result, combines long short-term memory knowledge and a target knowledge base to process the user's question-and-answer data to obtain answer data. The invention then controls the smart toy to output the answer data. By considering the user's emotions, this invention provides interactive content that better matches the user's emotions, solving the problems of existing smart toys having limited voice interaction methods, failing to respond appropriately to a child's current emotional state, and resulting in a poor companionship experience, thus enhancing the user's interactive experience.

[0004] In a first aspect, embodiments of the present invention provide a control method for an intelligent toy, the method comprising the following steps:

[0005] When the smart toy is in a question-and-answer state, it acquires the user's voice data and / or image data;

[0006] The user's emotion recognition result is determined by performing emotion recognition processing on the voice data and / or image data using a preset emotion recognition model.

[0007] Based on the emotion recognition results, combined with long short-term memory knowledge and the target knowledge base, the user's question and answer data is processed to obtain answer data. The level of the target knowledge base corresponds to the current growth value of the smart toy. Different levels of the target knowledge base have knowledge data corresponding to different user ages.

[0008] Control the smart toy to output the answer data.

[0009] Optionally, before acquiring the user's voice data and / or image data when the smart toy is in a question-and-answer state, the method further includes:

[0010] When the smart toy is activated, it acquires the user's initial voice data.

[0011] Based on the first voice data, determine whether the smart toy is in a question-and-answer state;

[0012] If the smart toy is in a question-and-answer state, then it acquires second voice data and / or image data.

[0013] Optionally, before performing emotion recognition processing on the voice data and / or image data using a preset emotion recognition model to determine the user's emotion recognition result, the method further includes:

[0014] Obtain a speech training dataset and a pre-trained emotion recognition model. The training dataset includes sample speech data and emotion annotation data corresponding to the speech data.

[0015] The pre-trained emotion recognition model is trained using the speech training dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using a minimum loss function. Once the training is complete, a preset first emotion recognition model is obtained. The first emotion recognition model is used to perform emotion recognition processing on the speech data.

[0016] Alternatively, obtain a text dataset and a pre-trained emotion recognition model, wherein the text dataset is obtained by speech-to-text processing based on a speech training dataset;

[0017] The pre-trained emotion recognition model is trained using the text dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using the minimum loss function. After training is completed, a preset second emotion recognition model is obtained. The second emotion recognition model is used to perform emotion recognition processing on the text data obtained after the speech data has been processed into text.

[0018] Alternatively, obtain an image training dataset and a pre-trained emotion recognition model, wherein the image training dataset includes sample image data and emotion annotation data corresponding to the sample image data;

[0019] The pre-trained emotion recognition model is trained using the image training dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using the minimum loss function. After training is completed, a preset third emotion recognition model is obtained. The third emotion recognition model is used to perform emotion recognition processing on the image data.

[0020] Alternatively, obtain at least two of the following: a speech training dataset, a text training dataset, and an image training dataset, as well as a pre-trained emotion recognition model. The speech training dataset includes sample speech data and emotion annotation data corresponding to the sample speech data. The image training dataset includes sample image data and emotion annotation data corresponding to the sample image data.

[0021] The pre-trained emotion recognition model is trained using at least two of the speech training dataset, the text dataset, and the image training dataset. During training, the parameters of the pre-trained emotion recognition model are adjusted using a minimum loss function. Upon completion of training, a preset fourth emotion recognition model is obtained. The fourth emotion recognition model is used to perform emotion recognition processing on at least two of the speech data, the text data, and the image data.

[0022] Optionally, the step of performing emotion recognition processing on the voice data and / or image data using a preset emotion recognition model to obtain the emotion recognition result of the target user includes:

[0023] The voice data is processed by a preset first emotion recognition model to obtain the voice emotion recognition result.

[0024] Alternatively, the voice data can be converted into text data, and the text data can be processed for emotion recognition using a preset second emotion recognition model to obtain the text emotion recognition result.

[0025] Alternatively, the image data can be processed using a pre-defined third emotion recognition model to obtain the image emotion recognition result;

[0026] Alternatively, by using a preset fourth emotion recognition model, emotion recognition processing can be performed on at least two of the voice data, text data, and image data to obtain multimodal emotion recognition results;

[0027] Based on the voice emotion recognition result, or the text emotion recognition result, or the image emotion recognition result, or the multimodal emotion recognition result, the emotion recognition result of the target user is obtained.

[0028] Optionally, before processing the user's question-and-answer data based on the emotion recognition results, combined with long short-term memory knowledge and the target knowledge base, to obtain the answer data, the method further includes:

[0029] Acquire the user's short-term memory data, which includes the user's historical dialogue records, user intent, and user emotional state data within a first preset time period;

[0030] Based on the aforementioned short-term memory data, short-term memory knowledge is constructed;

[0031] Acquire the user's long-term memory data, which includes the user's profile data and event data within a second preset time period, the second preset time period being longer than the first preset time period;

[0032] Based on the aforementioned long-term memory data, long-term memory knowledge is constructed.

[0033] Optionally, based on the emotion recognition results, and in conjunction with long short-term memory knowledge and a target knowledge base, the user's question-and-answer data is processed to obtain answer data, including:

[0034] Based on the emotion recognition results and long short-term memory knowledge, a corresponding emotion processing strategy is generated in the target knowledge base;

[0035] Based on the aforementioned emotion processing strategy, the question-and-answer data is processed to obtain answer data.

[0036] Optionally, the step of controlling the smart toy to output the answer data includes:

[0037] Based on the semantics of the answer data, the output mode of the smart toy is determined to be voice mode and / or action mode;

[0038] Based on the voice pattern and / or action pattern, the smart toy is controlled to output the answer data via voice and / or action.

[0039] Secondly, embodiments of the present invention also provide a control device for a smart toy, the control device for the smart toy comprising:

[0040] The first acquisition module is used to acquire the user's voice data and / or image data when the smart toy is in a question-and-answer state;

[0041] The first processing module is used to perform emotion recognition processing on the voice data and / or image data through a preset emotion recognition model to determine the emotion recognition result of the user.

[0042] The second processing module is used to process the user's question-and-answer data based on the emotion recognition results, combined with long short-term memory knowledge and the target knowledge base, to obtain answer data. The level of the target knowledge base corresponds to the current growth value of the smart toy, and different levels of the target knowledge base have knowledge data corresponding to different user ages.

[0043] The control module is used to control the smart toy to output the answer data.

[0044] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the control method for the intelligent toy provided in the embodiments of the present invention.

[0045] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the control method for the intelligent toy provided in the embodiments of the present invention.

[0046] In this embodiment of the invention, when the smart toy is in a question-and-answer state, it acquires the user's voice data and / or image data; it performs emotion recognition processing on the voice data and / or image data using a preset emotion recognition model to determine the user's emotion recognition result; based on the emotion recognition result, combined with long short-term memory knowledge and a target knowledge base, it processes the user's question-and-answer data to obtain answer data. The level of the target knowledge base corresponds to the current growth value of the smart toy, and different levels of the target knowledge base have knowledge data corresponding to different user ages; the smart toy is then controlled to output the answer data. This invention, by considering the user's emotions, provides interactive content that better matches the user's emotions, enhancing the user's interactive experience. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart of a control method for an intelligent toy provided in an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the structure of a control device for an intelligent toy provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] like Figure 1 As shown, Figure 1 This is a flowchart of a control method for an intelligent toy provided in an embodiment of the present invention. The control method for the intelligent toy includes the following steps:

[0053] 101. When the smart toy is in question-and-answer mode, acquire the user's voice data and / or image data.

[0054] In this embodiment of the invention, the control method for the smart toy can be applied to a server, with a communication connection between the server and the smart toy. The smart toy includes a voice acquisition device and an image acquisition device; the voice acquisition device is used to acquire voice data, and the image acquisition device is used to acquire image data. Upon initial use of the smart toy, the user can set user information, including the user's name, age, hobbies, etc.; the user can set a wake word for the smart toy; and the smart toy can be configured with a special nickname for a specific user.

[0055] The users mentioned above can be understood as those who wake up the smart toy.

[0056] The above question-and-answer state can be understood as the state in which the smart toy can interact with the user through question-and-answer sessions.

[0057] The aforementioned voice data is acquired in real time through the voice acquisition device of the intelligent toy, and includes the user's tone of voice, volume, etc. The aforementioned image data is acquired in real time through the intelligent toy's image acquisition device, and includes images of the user's facial expressions, such as happy, sad, angry, etc.

[0058] It should be noted that when using smart toys, users can set their age to M years old, where M can be 3, 4, 5, etc. The smart toy can match the user's set age with the corresponding level of knowledge base. Different levels of the knowledge base correspond to knowledge data for different ages. The smart toy can interact with the user through the knowledge data of the corresponding level of knowledge base, which effectively improves the companionship experience of the smart toy.

[0059] 102. Perform emotion recognition processing on voice data and / or image data using a preset emotion recognition model to determine the user's emotion recognition result.

[0060] In this embodiment of the invention, the aforementioned preset emotion recognition model can be an emotion recognition model built based on deep learning or machine learning, such as a Long Short-Term Memory Network (LSTM) or a Gated Recurrent Unit (GRU). The emotion recognition model can automatically detect and recognize human emotional states, such as happiness, sadness, and anger.

[0061] The aforementioned preset emotion recognition models include a first emotion recognition model, a second emotion recognition model, a third emotion recognition model, and a fourth emotion recognition model.

[0062] The aforementioned first emotion recognition model is obtained by training an untrained emotion recognition model using a speech training dataset. This training dataset includes a large amount of sample speech data and corresponding emotion annotations. Using this first emotion recognition model, the user's emotional state can be automatically identified based on the speech data.

[0063] The aforementioned second emotion recognition model trains the untrained emotion recognition model using a text training dataset. Once training is complete, the second emotion recognition model is obtained. This text training dataset can be obtained by converting the speech training dataset to text, or it can be a text training dataset for emotion recognition downloaded from an open-source database. The text training dataset includes a large amount of sample text data and corresponding emotion annotation data. Using this second emotion recognition model, the user's emotional state can be automatically identified based on the text data converted from speech data.

[0064] The aforementioned third emotion recognition model is obtained by training an untrained emotion recognition model using an image training dataset. This training dataset includes a large number of sample images and corresponding emotion annotations. Using this third emotion recognition model, the user's facial emotion state can be automatically identified based on image data.

[0065] The aforementioned fourth emotion recognition model is trained by using at least two of the following datasets—voice training dataset, text training dataset, and image training dataset—to train an untrained emotion recognition model. Once training is complete, the fourth emotion recognition model is obtained. The training datasets can be a combination of voice and image training datasets, a combination of voice and text training datasets, a combination of text and image training datasets, or a combination of all three. The voice training dataset includes a large amount of sample voice data and corresponding emotion annotations. The text training dataset includes a large amount of sample text data and corresponding emotion annotations. The image training dataset includes a large amount of sample image data and corresponding emotion annotations. Using this fourth emotion recognition model, the user's emotional state can be identified based on at least two of the following three datasets: voice data, text data, and image data.

[0066] The labeled data mentioned above can be understood as adding structured labels to the raw data, enabling the machine learning model to recognize and process both the raw and labeled data. Through labeled data, the model can establish a mapping between input data and the correct output label. The training can be supervised, which uses a set of data with known labels to train the model. By optimizing the model parameters, the model can predict the label of new data or make decisions based on the characteristics of existing data. During training, a minimum loss function can be used to adjust the model's parameters to minimize the difference between the model's output label and the input data. The loss function measures the difference between the model's prediction and the true result; its purpose is to improve prediction accuracy by minimizing the loss function value through adjusting the model parameters. The loss function mentioned above can be a mean squared error loss function, a cross-entropy loss function, etc. The untrained emotion recognition model mentioned above can be built based on machine learning or deep learning models, such as Long Short-Term Memory (LSTM) networks or Gated Recurrent Units (GRUs).

[0067] The aforementioned emotion recognition processing can be understood as the process of analyzing a user's voice data and / or image data to identify the emotional state expressed in the voice data and / or the emotional state expressed in the image data.

[0068] The aforementioned emotion recognition results are obtained through emotion recognition processing of speech data and / or image data, including speech emotion recognition results and / or image recognition results. These emotion recognition results can include happiness, sadness, anger, etc.

[0069] The aforementioned emotions refer to the experiences of attitudes towards external things that accompany cognitive and conscious processes, including joy, anger, sorrow, and happiness.

[0070] 103. Based on the emotion recognition results, combined with long and short-term memory knowledge and the target knowledge base, the user's question and answer data is processed to obtain the answer data.

[0071] In this embodiment of the invention, the level of the target knowledge base corresponds to the current growth value of the smart toy, and different levels of the target knowledge base have knowledge data corresponding to different user ages. The current growth value of the smart toy can be understood as the current growth value of the smart toy in interaction with the user, and the growth value of the smart toy corresponds to the level of the knowledge base.

[0072] The aforementioned knowledge of both long-term and short-term memory includes both. Long-term memory knowledge is used to record each user's personalized information, including personal details, interests, and other information. This long-term memory knowledge can be understood, updated, and enriched through continuous dialogue, revealing the user's personality. During user conversations, it can recall relevant memories and generate responses that match the context and user profile, making emotional companionship scenarios more realistic and personalized. It should be noted that long-term memory is isolated between users; each user uses their own memory content generated from their interaction with the smart toy.

[0073] The structure of the aforementioned short-term memory knowledge includes a dialogue history stack, an intent state machine, an emotion state queue, and a task list. The dialogue history stack can be understood as using a stack structure to store the most recent rounds of dialogue, each round containing user input and system response; the intent state machine can be understood as using a state machine model to manage changes in user intent, ensuring the system can flexibly respond to different dialogue scenarios; the emotion state queue can be understood as using a queue structure to record the user's recent emotion changes for trend analysis; and the task list can be a list structure that records user-assigned tasks and their states.

[0074] The aforementioned short-term memory knowledge records the user's current task (such as setting reminders, querying information, etc.), task progress, current time and location information, the user's recently expressed preferences and interests (such as recently liked songs, movies, etc.), and temporary parameters set by the user in the current conversation (such as volume, language preferences, etc.). Based on the current environment and situation, it adjusts the content and manner of the smart toy's dialogue with the user, and provides more specific and relevant suggestions.

[0075] The aforementioned long-term memory knowledge can automatically identify and record personalized information provided by users in the conversation, retrieve relevant long-term memories when requested by the user, summarize individual biochemical information, and generate the final answer data based on the summarized biochemical information and the user's current question and answer data.

[0076] The aforementioned target knowledge base can be understood as the knowledge base corresponding to the current growth value of the smart toy.

[0077] The above Q&A data can be understood as data containing user questions.

[0078] The above question-and-answer processing can be understood as the process of finding the most relevant answer to the question-and-answer data from the target knowledge base.

[0079] The above answer data is the most relevant answer data obtained by analyzing question and answer data, and the answer data is based on...

[0080] Furthermore, after identifying the user's current emotion, the system will combine the user's long-term memory knowledge and short-term memory knowledge to more accurately understand the user's emotional state.

[0081] In one possible implementation, for example, if a user frequently exhibits happy emotions but suddenly becomes sad at a certain moment, the system will understand the reasons for this emotional change based on the user's long-term and short-term memory knowledge, such as whether an unpleasant event has occurred, or whether the user's physical or psychological state has changed. Then, based on these reasons for the change, and combined with the corresponding age-related emotion question-and-answer processing methods in the target knowledge base, the system will process the user's question-and-answer data to obtain answer data. The answer data includes comforting words and suggestions to change the user's mood.

[0082] 104. Control the intelligent toy to output the answer data.

[0083] In this embodiment of the invention, the above-mentioned output can be understood as the output form of the smart toy, which can be either voice output or action output. Voice output can be understood as the smart toy converting answer data into audible voice information. Action output can be understood as the smart toy performing corresponding physical actions based on the answer data, such as moving its eyes, rotating its arms, or moving other parts. Through action output, users can obtain answers in a direct or interactive manner.

[0084] In one possible embodiment, when the smart toy's output mode is voice mode and action mode, the answer data is converted into audible voice information through voice, and the answer data is expressed through the smart toy's physical actions, such as moving the eyes, rotating the limbs, shaking the head, wagging the tail, etc.; when the smart toy's output mode is voice mode, the answer data is converted into audible voice information through voice; when the smart toy's output mode is action mode, the answer data is expressed through the smart toy's physical actions, such as moving the eyes, rotating the arms, or moving other parts.

[0085] In this embodiment of the invention, when the smart toy is in a question-and-answer state, it acquires the user's voice data and / or image data; it performs emotion recognition processing on the voice data and / or image data using a preset emotion recognition model to determine the user's emotion recognition result; based on the emotion recognition result, combined with long short-term memory knowledge and a target knowledge base, it processes the user's question-and-answer data to obtain answer data. The level of the target knowledge base corresponds to the current growth value of the smart toy, and different levels of the target knowledge base have knowledge data corresponding to different user ages; the smart toy is then controlled to output the answer data. This invention, by considering the user's emotions and providing interactive content that better matches the user's emotions, solves the problem that existing smart toys have a single voice interaction method, cannot provide responses that match the child's current emotional state, and result in a poor companionship experience, thus enhancing the user's interactive experience.

[0086] It is understood that in the specific implementation of this application, data such as voice data, image data, knowledge data, and user data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0087] Optionally, before the step of acquiring the target user's voice data and / or image data when a question-and-answer intent is detected, the user's first voice data is acquired after the smart toy is woken up; based on the first voice data, it is determined whether the smart toy is in a question-and-answer state; if the smart toy is in a question-and-answer state, then second voice data and / or image data are acquired.

[0088] In this embodiment of the invention, the user can set a wake word for the smart toy when it is used for the first time.

[0089] The aforementioned wake-up process can be understood as the process by which a user activates a smart toy in a specific way. Users can wake up the smart toy using a set wake word, or through touch wake-up operations, etc.

[0090] The users mentioned above can be understood as those who wake up the smart toy.

[0091] The aforementioned first voice data may contain data related to questions, such as "I want to know" or "Can you tell me?"

[0092] The aforementioned second voice data includes the first voice data. The second voice data can be understood as question-and-answer data, which is data about questions, such as "I want to know XX" or "Can you tell me XX?"

[0093] The above question-and-answer state can be understood as the state in which the smart toy can interact with the user through question-and-answer sessions.

[0094] Specifically, when a smart toy is activated by a user's wake word, it acquires the user's initial voice data and performs semantic recognition on it to determine the user's intent. This user intent can be understood as the purpose the user wants to express or the action they want to perform, such as a question-and-answer intent or a feeding intent. The semantic recognition process involves processing the voice data semantically, which can be done using natural language processing (NLP) technology. NLP technology enables machines to understand, interpret, and generate human language, facilitating effective human-machine communication and enabling computers to perform tasks such as language translation, sentiment analysis, and text summarization.

[0095] Furthermore, when the user's intent is a question-and-answer intent, the smart toy is determined to be in a question-and-answer state; when the user's intent is a feeding intent, the smart toy is not in a question-and-answer state.

[0096] Optionally, before the step of performing emotion recognition processing on speech data and / or image data using a preset emotion recognition model to determine the user's emotion recognition result, a training dataset and a pre-trained emotion recognition model can be obtained; the pre-trained emotion recognition model can be trained using the training dataset, and during the training process, the parameters of the pre-trained emotion recognition model can be adjusted using a minimum loss function. After training is completed, a preset first emotion recognition model is obtained, which is used to perform emotion recognition processing on speech data; or, a text dataset and a pre-trained emotion recognition model can be obtained, where the text dataset is obtained by performing speech-to-text processing on the speech training dataset; the pre-trained emotion recognition model can be trained using the text dataset, and during the training process, the parameters of the pre-trained emotion recognition model can be adjusted using a minimum loss function. After training is completed, a preset second emotion recognition model is obtained, which is used to perform emotion recognition processing on the text data obtained after speech-to-text processing on the speech data; or, an image training dataset and a pre-trained... A pre-trained emotion recognition model is obtained by training the model using an image training dataset. During training, the parameters of the pre-trained emotion recognition model are adjusted using a minimum loss function. After training, a preset third emotion recognition model is obtained, which is used to perform emotion recognition processing on the image data. Alternatively, at least two of the following datasets are obtained: a speech training dataset, a text training dataset, and an image training dataset, along with a pre-trained emotion recognition model. The speech training dataset includes sample speech data and corresponding emotion annotation data. The image training dataset includes sample image data and corresponding emotion annotation data. The pre-trained emotion recognition model is trained using at least two of the following datasets: a speech training dataset, a text dataset, and an image training dataset. During training, the parameters of the pre-trained emotion recognition model are adjusted using a minimum loss function. After training, a preset fourth emotion recognition model is obtained, which is used to perform emotion recognition processing on at least two of the speech data, text data, and image data.

[0097] In this embodiment of the invention, the above-mentioned speech training dataset includes a large amount of sample speech data and emotion annotation data corresponding to the sample speech data.

[0098] The aforementioned text training dataset includes a large amount of sample text data and corresponding sentiment annotation data for the sample text data.

[0099] The aforementioned image training dataset includes sample image data and corresponding emotion annotation data for the sample image data.

[0100] The aforementioned pre-trained emotion recognition model can be an emotion recognition model built based on deep learning or machine learning, such as a Long Short-Term Memory Network (LSTM), a Gated Recurrent Unit (GRU), etc.

[0101] The training described above can be supervised training. Supervised training uses a set of data with known labels to train the model. By optimizing the model parameters, the model can predict the labels of new data or make decisions based on the characteristics of existing data.

[0102] The loss function described above measures the difference between the model's predictions and the actual results. By minimizing the loss function, the model's parameters can be optimized, making the model's predictions more accurate and thus improving the model's prediction accuracy. The loss function can be the mean squared error loss function, cross-entropy loss function, etc.

[0103] The above parameter adjustments can be understood as the process of optimizing model performance and prediction accuracy by adjusting the model's hyperparameters and model parameters.

[0104] Specifically, sample speech data from the speech training dataset can be input into a pre-trained emotion recognition model for emotion recognition processing to obtain speech emotion recognition results corresponding to the sample speech data. A loss function is used to calculate the speech emotion recognition results corresponding to the sample speech data and the corresponding emotion annotation data to obtain the error loss value between the speech emotion recognition results corresponding to the sample speech data and the corresponding emotion annotation data. The optimization objective is to minimize this error loss value. The parameters of the pre-trained emotion recognition model are adjusted through the backpropagation algorithm. The above parameter adjustment process is iterated. When the error loss value is less than the preset value, or the number of iterations reaches the preset number, the iteration stops, the training is completed, and the preset first emotion recognition model is obtained.

[0105] Sample text data from the text training dataset can be input into a pre-trained emotion recognition model for emotion recognition processing, resulting in text emotion recognition results corresponding to the sample text data. A loss function is used to calculate the error loss value between the text emotion recognition results and the corresponding emotion annotation data. Minimizing this error loss value is the optimization objective. The parameters of the pre-trained emotion recognition model are adjusted using a backpropagation algorithm. The parameter adjustment process is iterated until the error loss value is less than a preset value or the number of iterations reaches a preset number. Then, the iteration stops, training is completed, and a preset second emotion recognition model is obtained.

[0106] Sample image data from the image training dataset can be input into a pre-trained emotion recognition model for emotion recognition processing, resulting in image emotion recognition results corresponding to the sample image data. A loss function is used to calculate the error loss value between the image emotion recognition results and the corresponding emotion annotation data of the sample image data. Minimizing this error loss value is the optimization objective. The parameters of the pre-trained emotion recognition model are adjusted using a backpropagation algorithm. The above parameter adjustment process is iterated. When the error loss value is less than a preset value, or the number of iterations reaches a preset number, the iteration stops, the training is completed, and the preset third emotion recognition model is obtained.

[0107] At least two of the following can be combined to form multimodal sample data: sample speech data from the speech training dataset, sample text data from the text training dataset, and sample image data from the image training dataset. The labeled data of the multimodal sample data is the union or intersection of the labeled data corresponding to the at least two sample data that make up the multimodal data. The multimodal sample data is then input into a pre-trained emotion recognition model for emotion recognition processing to obtain the multimodal emotion recognition result corresponding to the multimodal sample data. A loss function is used to calculate the error loss value between the multimodal emotion recognition result and the corresponding emotion labeled data. The optimization objective is to minimize this error loss value. The parameters of the pre-trained emotion recognition model are adjusted using a backpropagation algorithm. The above parameter adjustment process is iterated. When the error loss value is less than a preset value or the number of iterations reaches a preset number, the iteration stops, the training is completed, and the preset fourth emotion recognition model is obtained.

[0108] After training, the aforementioned preset first emotion recognition model can automatically recognize the user's voice emotion state based on voice data, the aforementioned preset second emotion recognition model can automatically recognize the user's text emotion state based on text data, the aforementioned preset third emotion recognition model can automatically recognize the user's facial emotion state based on image data, and the aforementioned preset fourth emotion recognition model can automatically recognize the user's multimodal emotion state based on at least two of the voice data, text data, and image data.

[0109] The aforementioned emotional states can include happiness, sadness, anger, etc.

[0110] Optionally, in the step of performing emotion recognition processing on voice data and / or image data using a preset emotion recognition model to obtain the emotion recognition result of the target user, the voice data can be processed using a preset first emotion recognition model to obtain a voice emotion recognition result; or, the voice data can be converted into text data, and the text data can be processed using a preset second emotion recognition model to obtain a text emotion recognition result; or, the image data can be processed using a preset third emotion recognition model to obtain an image emotion recognition result; or, at least two of the voice data, text data, and image data can be processed using a preset fourth emotion recognition model to obtain a multimodal emotion recognition result; and the emotion recognition result of the target user is obtained based on the voice emotion recognition result, or the text emotion recognition result, or the image emotion recognition result, or the multimodal emotion recognition result.

[0111] In this embodiment of the invention, the preset emotion recognition model may be a first emotion recognition model, a second emotion recognition model, a third emotion recognition model, or a fourth emotion recognition model. The preset first emotion recognition model can automatically recognize the user's voice emotion state based on voice data. The preset second emotion recognition model can automatically recognize the user's text emotion state based on text data. The preset third emotion recognition model can automatically recognize the user's facial emotion state based on image data. The preset fourth emotion recognition model can automatically recognize the user's multimodal emotion state based on at least two of the voice data, text data, and image data.

[0112] The aforementioned first emotion recognition model is obtained by training an untrained emotion recognition model using a speech training dataset. This training dataset includes a large amount of sample speech data and corresponding emotion annotation data.

[0113] The aforementioned second emotion recognition model is obtained by training the untrained emotion recognition model using a text training dataset. This training dataset includes a large amount of sample text data and corresponding emotion annotations.

[0114] The aforementioned third emotion recognition model is obtained by training an untrained emotion recognition model using an image training dataset. This training dataset includes a large number of sample images and corresponding emotion annotations.

[0115] The aforementioned fourth emotion recognition model is obtained by training an untrained emotion recognition model using at least two of the following datasets: a speech training dataset, a text training dataset, and an image training dataset. The training datasets include at least two of the following: a speech training dataset, a text training dataset, and an image training dataset. The speech training dataset includes a large amount of sample speech data and corresponding emotion annotation data. The image training dataset includes a large amount of sample image data and corresponding emotion annotation data.

[0116] The above voice emotion recognition results are obtained by performing emotion recognition on voice data. The above voice emotion recognition results can be happy emotions, sad emotions, etc.

[0117] The above text emotion recognition results are obtained by performing emotion recognition on text data. The above text emotion recognition results can be happy emotions, sad emotions, etc.

[0118] The above image emotion recognition results are obtained by performing emotion recognition on image data. The above image emotion recognition results can be happy, sad, etc.

[0119] The above-mentioned multimodal emotion recognition results are obtained by performing emotion recognition on at least two of the voice data, text data, and image data. The above-mentioned multimodal emotion recognition results can be happy states, sad states, etc.

[0120] It should be noted that speech data can be encoded to obtain encoded speech data, and then a preset emotion recognition model can be used to analyze the speech emotion of the encoded semantic data to obtain the speech emotion recognition result. Alternatively, speech data can be converted to text to obtain text data, and then a preset emotion recognition model can be used to analyze the text emotion of the text data to obtain the speech emotion recognition result. The speech encoding process described above can be understood as the process of converting speech data signals into digital signals. The purpose of speech encoding is to use as little communication capacity as possible to transmit the highest possible quality speech processing. The text conversion process described above can be understood as the process of converting speech data into text data.

[0121] In one possible embodiment, the speech data can be processed by a first emotion recognition model to obtain a speech emotion recognition result; the text data can be processed by a second emotion recognition result to obtain a text emotion recognition result; the image data can be processed by a third emotion recognition result to obtain an image emotion recognition result; and at least two of the speech data, text data, and image data can be processed by a fourth emotion recognition model to obtain a multimodal emotion recognition result.

[0122] In one possible implementation, a multimodal emotion recognition model (such as Emotion-LLaMA, MER, etc.) can be fine-tuned using a speech training dataset to obtain a first emotion recognition model. Alternatively, a multimodal emotion recognition model can be fine-tuned using a text training dataset to obtain a second emotion recognition model. Alternatively, a multimodal emotion recognition model can be fine-tuned using an image training dataset to obtain a third emotion recognition model. Alternatively, a multimodal emotion recognition model can be fine-tuned using at least two of the speech, text, and image training datasets to obtain a fourth emotion recognition model.

[0123] Optionally, before the step of processing the user's question-and-answer data based on emotion recognition results, combined with long and short-term memory knowledge and the target knowledge base, to obtain answer data, it is also possible to obtain the user's short-term memory data; construct short-term memory knowledge based on the short-term memory data; obtain the user's long-term memory data; and construct long-term memory knowledge based on the long-term memory data.

[0124] In this embodiment of the invention, the aforementioned short-term memory data includes the user's historical dialogue records, user intent, and user emotional state data within a first preset time period. The first preset time period can be understood as a recent period, such as the most recent day, the most recent two days, etc. The aforementioned historical dialogue records can be understood as the history of the most recent rounds of dialogue, each round including user input and system response. The aforementioned user intent can be understood as changes in the user's intent within the dialogue records. The aforementioned user emotional state data includes the user's recent emotional changes.

[0125] The structure of the aforementioned short-term memory knowledge includes a dialogue history stack, an intent state machine, an emotion state queue, and a task list. The dialogue history stack can be understood as using a stack structure to store the most recent rounds of dialogue, each round containing user input and system response; the intent state machine can be understood as using a state machine model to manage changes in user intent, ensuring the system can flexibly respond to different dialogue scenarios; the emotion state queue can be understood as using a queue structure to record the user's recent emotion changes for trend analysis; and the task list can be a list structure that records user-assigned tasks and their states.

[0126] The aforementioned short-term memory knowledge records the user's current task (such as setting reminders, querying information, etc.), task progress, current time and location information, the user's recently expressed preferences and interests (such as recently liked songs, movies, etc.), and temporary parameters set by the user in the current conversation (such as volume, language preferences, etc.). Based on the current environment and situation, it adjusts the content and manner of the smart toy's dialogue with the user, and provides more specific and relevant suggestions.

[0127] The aforementioned long-term memory data includes user profile data and event data within a second preset time period, which is longer than the first preset time period. The second preset time period can be understood as the total time the user interacts with the smart toy, from the initial setup to the current usage period. The aforementioned user profile data can be understood as the user's personal information and preferences, including age, gender, and personal preferences. The aforementioned event data can be understood as a key event that occurred on a specific date, such as the user scoring 100 on yesterday's exam or drinking milk this morning. The aforementioned long-term memory knowledge can record the user's personal preferences, special dates, close relationships, learning and development, health status, emotional support, and play history. The aforementioned personal preferences could be the user's favorite books, movies, music, etc.; the aforementioned special days could be the user's birthday or an anniversary; the aforementioned close relationships could be understood as the user's close relationships with family members, friends, etc.; the aforementioned learning and development could be understood as information related to the user's learning, such as scoring 100 points on a recent Chinese exam; the aforementioned health status could be understood as paying attention to the user's sleep quality, exercise habits, and other lifestyle factors; the aforementioned emotional support could be understood as recording the user's emotional changes; and the aforementioned game records could be understood as the games the user has played.

[0128] It's worth noting that smart toys can send blessings or reminders to users on their birthdays or anniversaries. Smart toys can also provide healthy living suggestions based on the user's health condition. Furthermore, smart toys can sense changes in the user's emotions and offer timely emotional comfort and support.

[0129] The aforementioned long-term memory knowledge is used to record each user's personalized information, including personal information, hobbies, and other details. This long-term memory knowledge can be understood, refreshed, and enriched through continuous dialogue, revealing the user's personality. During user conversations, it can recall relevant memories and generate responses that match the context and user profile, making emotional companionship scenarios more realistic and personalized. It should be noted that long-term memories are isolated between users; each user uses their own memories generated from conversations with the smart toy.

[0130] The aforementioned long-term memory knowledge can automatically identify and record personalized information provided by users in conversations, retrieve relevant long-term memories upon user request, summarize individual biochemical information, and generate final answer data based on the summarized personalized information and the user's current question and answer data.

[0131] Optionally, in the step of processing user question-and-answer data based on emotion recognition results, combined with long short-term memory knowledge and target knowledge base, to obtain answer data, a corresponding emotion processing strategy can be generated in the target knowledge base based on emotion recognition results and long short-term memory knowledge; and the question-and-answer data can be processed based on the emotion processing strategy to obtain answer data.

[0132] In this embodiment of the invention, the emotion recognition result can be happy, sad, angry, etc.

[0133] The aforementioned target knowledge base can be understood as the knowledge base corresponding to the current growth value of the intelligent toy. This knowledge base is...

[0134] The aforementioned long and short-term memory knowledge includes both long-term and short-term memory knowledge. The long-term memory knowledge can automatically identify and record personalized information provided by the user during conversations, retrieve relevant long-term memories upon user request, summarize personalized information, and generate final answer data based on this summary and the user's current question-and-answer data. The short-term memory knowledge records the user's current task, task progress, current time and location, recently expressed preferences and interests, and temporary parameters set by the user in the current conversation. Based on the current environment and situation, it adjusts the content and manner of the smart toy's dialogue with the user, and provides more specific and relevant suggestions.

[0135] The aforementioned emotion processing strategy is a method and rule for adjusting or optimizing problem-solving based on the user's emotion recognition results and combined with long and short-term memory knowledge. This strategy ensures that the generated responses are not only accurate in content but also match the user's emotional expression, thereby improving user experience and satisfaction.

[0136] The above question-and-answer data refers to data that includes questions.

[0137] The above answer data is the most relevant answer data obtained by analyzing question and answer data.

[0138] In one possible implementation, when a user is in a happy mood, the style and content of the answer data can be adjusted based on the user's past emotional history and preferences, using a more positive and encouraging language style to interact with the user and make it more in line with the user's personal preferences; if the user is frustrated or dissatisfied, the answer data can reflect understanding and support for the user's emotions based on the user's past emotional history and preferences, avoiding mentioning unpleasant topics.

[0139] Optionally, in the step of controlling the smart toy to output answer data, the output mode of the smart toy can be determined to be a voice mode and / or an action mode based on the semantics of the answer data; based on the voice mode and / or action mode, the smart toy can be controlled to output the answer data via voice and / or action.

[0140] In this embodiment of the invention, after obtaining the answer data, semantic recognition can be performed on the answer data to identify its semantics. Natural language processing (NLP) techniques can be used to analyze the word meanings and semantics of the answer data to understand its meaning and content. The aforementioned NLP techniques aim to enable machines to understand, interpret, and generate human language, achieving effective communication between humans and machines, and enabling computers to perform tasks such as language translation, sentiment analysis, and text summarization.

[0141] Specifically, a semantic classification model can be trained to predict the type of output pattern based on the semantics of the answer data. This involves collecting sample data and corresponding output pattern labels, constructing an initial semantic classification model based on a neural network or support vector machine, inputting the sample semantic data into the initial model, and calculating the error loss between the predicted output pattern and the output pattern label. The model parameters are then adjusted iteratively, with the goal of minimizing the error loss, until the error loss is less than the predicted value or the number of iterations reaches the predicted number. This results in a trained semantic classification model. The answer data is then input into this model to obtain the output pattern classification. The output of this semantic classification model represents the output pattern classification, which includes speech output and / or action output. The sample data mentioned above represents the collected answer data.

[0142] The semantics of the above answer data can be understood as the meaning and content expressed by the answer data.

[0143] The above output mode can be understood as the way smart toys interact with users.

[0144] The above-mentioned voice mode can be understood as an output mode that conveys information through voice.

[0145] The above-mentioned action patterns can be understood as output patterns that express information through physical actions, such as head shaking, limb rotation, and other actions.

[0146] The aforementioned voice output can be understood as the smart toy converting answer data into audible voice information through voice.

[0147] The above-mentioned action output can be understood as the intelligent toy performing corresponding physical actions based on the answer data, such as shaking its head or rotating its limbs.

[0148] Furthermore, when the smart toy's output mode is voice mode and action mode, the answer data is converted into audible voice information through voice, and the answer data is expressed through the smart toy's physical actions; when the smart toy's output mode is voice mode, the answer data is converted into audible voice information through voice; when the smart toy's output mode is action mode, the answer data is expressed through the smart toy's physical actions.

[0149] like Figure 2 As shown, an embodiment of the present invention provides a control device for a smart toy, the control device for the smart toy comprising:

[0150] The first acquisition module 201 is used to acquire the user's voice data and / or image data when the smart toy is in a question-and-answer state;

[0151] The first processing module 202 is used to perform emotion recognition processing on the voice data and / or image data through a preset emotion recognition model to determine the emotion recognition result of the user.

[0152] The second processing module 203 is used to process the user's question and answer data based on the emotion recognition results, combined with long short-term memory knowledge and the target knowledge base, to obtain answer data. The level of the target knowledge base corresponds to the current growth value of the smart toy, and different levels of the target knowledge base have knowledge data corresponding to different user ages.

[0153] The control module 204 is used to control the smart toy to output the answer data.

[0154] Optionally, the device further includes:

[0155] The second acquisition module is used to acquire the user's first voice data after the smart toy is woken up;

[0156] The first determining module is used to determine whether the smart toy is in a question-and-answer state based on the first voice data;

[0157] The third acquisition module is used to acquire second voice data and / or image data if the smart toy is in a question-and-answer state.

[0158] Optionally, the device further includes:

[0159] The fourth acquisition module is used to acquire a speech training dataset and a pre-trained emotion recognition model. The training dataset includes sample speech data and emotion annotation data corresponding to the speech data.

[0160] The first training module is used to train the pre-trained emotion recognition model using the speech training dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using the minimum loss function. After training is completed, a preset first emotion recognition model is obtained. The first emotion recognition model is used to perform emotion recognition processing on the speech data.

[0161] Alternatively, the fifth acquisition module is used to acquire a text dataset and a pre-trained emotion recognition model, wherein the text dataset is obtained by speech-to-text processing based on the speech training dataset;

[0162] The second training module is used to train the pre-trained emotion recognition model using the text dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using the minimum loss function. After training is completed, a preset second emotion recognition model is obtained. The second emotion recognition model is used to perform emotion recognition processing on the text data obtained after the speech data has been processed into text.

[0163] Alternatively, the sixth acquisition module is used to acquire an image training dataset and a pre-trained emotion recognition model, wherein the image training dataset includes sample image data and emotion annotation data corresponding to the sample image data;

[0164] The third training module is used to train the pre-trained emotion recognition model using the image training dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using the minimum loss function. After training is completed, a preset third emotion recognition model is obtained. The third emotion recognition model is used to perform emotion recognition processing on the image data.

[0165] Alternatively, the seventh acquisition module is used to acquire at least two of the following: a speech training dataset, a text training dataset, and an image training dataset, as well as a pre-trained emotion recognition model. The speech training dataset includes sample speech data and emotion annotation data corresponding to the sample speech data. The image training dataset includes sample image data and emotion annotation data corresponding to the sample image data.

[0166] The fourth training module is used to train the pre-trained emotion recognition model using at least two of the speech training dataset, the text dataset, and the image training dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using a minimum loss function. After training is completed, a preset fourth emotion recognition model is obtained. The fourth emotion recognition model is used to perform emotion recognition processing on at least two of the speech data, the text data, and the image data.

[0167] Optionally, the first processing module 202 includes:

[0168] The first processing submodule is used to perform emotion recognition processing on the voice data through a preset first emotion recognition model to obtain the voice emotion recognition result.

[0169] Alternatively, the second processing submodule is used to convert the voice data into text data, and perform emotion recognition processing on the text data through a preset second emotion recognition model to obtain the text emotion recognition result;

[0170] Alternatively, a third processing submodule is used to perform emotion recognition processing on the image data through a preset third emotion recognition model to obtain the image emotion recognition result;

[0171] Alternatively, the fourth processing submodule is used to perform emotion recognition processing on at least two of the voice data, text data, and image data using a preset fourth emotion recognition model to obtain a multimodal emotion recognition result;

[0172] The fifth processing submodule is used to obtain the emotion recognition result of the target user based on the voice emotion recognition result, or the text emotion recognition result, or the image emotion recognition result, or the multimodal emotion recognition result.

[0173] Optionally, the device further includes:

[0174] The eighth acquisition module is used to acquire the user's short-term memory data, which includes the user's historical dialogue records, user intent, and user emotional state data within a first preset time period.

[0175] The first construction module is used to construct short-term memory knowledge based on the short-term memory data;

[0176] The ninth acquisition module is used to acquire the user's long-term memory data, which includes the user's profile data and event data within a second preset time period, the second preset time period being longer than the first preset time period;

[0177] The second construction module is used to construct long-term memory knowledge based on the long-term memory data.

[0178] Optionally, the second processing module includes:

[0179] A generation submodule is used to generate corresponding emotion processing strategies in the target knowledge base based on the emotion recognition results and long short-term memory knowledge.

[0180] The fourth processing submodule is used to perform question-and-answer processing on the question-and-answer data based on the emotion processing strategy to obtain answer data.

[0181] Optionally, the control module 204 includes:

[0182] The determination submodule is used to determine, based on the semantics of the answer data, whether the output mode of the smart toy is a voice mode and / or an action mode;

[0183] The control submodule is used to control the smart toy to output voice and / or action data based on the voice mode and / or action mode.

[0184] like Figure 3 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described control methods for intelligent toys.

[0185] Specifically, it includes a processor 301 and a memory 302, as well as a computer program stored in the memory 302 and capable of running on the processor 301, which executes the control method for the intelligent toy, wherein:

[0186] The processor 301 executes the calculator program for the control method of the intelligent toy stored in the memory 302, and performs the following steps:

[0187] When the smart toy is in a question-and-answer state, it acquires the user's voice data and / or image data;

[0188] The user's emotion recognition result is determined by performing emotion recognition processing on the voice data and / or image data using a preset emotion recognition model.

[0189] Based on the emotion recognition results, combined with long short-term memory knowledge and the target knowledge base, the user's question and answer data is processed to obtain answer data. The level of the target knowledge base corresponds to the current growth value of the smart toy. Different levels of the target knowledge base have knowledge data corresponding to different user ages.

[0190] Control the smart toy to output the answer data.

[0191] Optionally, before acquiring the user's voice data and / or image data when the smart toy is in a question-and-answer state, the method executed by the processor 301 further includes:

[0192] When the smart toy is activated, it acquires the user's initial voice data.

[0193] Based on the first voice data, determine whether the smart toy is in a question-and-answer state;

[0194] If the smart toy is in a question-and-answer state, then it acquires second voice data and / or image data.

[0195] Optionally, before performing emotion recognition processing on the voice data and / or image data using a preset emotion recognition model to determine the user's emotion recognition result, the method executed by the processor 301 further includes:

[0196] Obtain a speech training dataset and a pre-trained emotion recognition model. The training dataset includes sample speech data and emotion annotation data corresponding to the speech data.

[0197] The pre-trained emotion recognition model is trained using the speech training dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using a minimum loss function. Once the training is complete, a preset first emotion recognition model is obtained. The first emotion recognition model is used to perform emotion recognition processing on the speech data.

[0198] Alternatively, obtain a text dataset and a pre-trained emotion recognition model, wherein the text dataset is obtained by speech-to-text processing based on a speech training dataset;

[0199] The pre-trained emotion recognition model is trained using the text dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using the minimum loss function. After training is completed, a preset second emotion recognition model is obtained. The second emotion recognition model is used to perform emotion recognition processing on the text data obtained after the speech data has been processed into text.

[0200] Alternatively, obtain an image training dataset and a pre-trained emotion recognition model, wherein the image training dataset includes sample image data and emotion annotation data corresponding to the sample image data;

[0201] The pre-trained emotion recognition model is trained using the image training dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using the minimum loss function. After training is completed, a preset third emotion recognition model is obtained. The third emotion recognition model is used to perform emotion recognition processing on the image data.

[0202] Alternatively, obtain at least two of the following: a speech training dataset, a text training dataset, and an image training dataset, as well as a pre-trained emotion recognition model. The speech training dataset includes sample speech data and emotion annotation data corresponding to the sample speech data. The image training dataset includes sample image data and emotion annotation data corresponding to the sample image data.

[0203] The pre-trained emotion recognition model is trained using at least two of the speech training dataset, the text dataset, and the image training dataset. During training, the parameters of the pre-trained emotion recognition model are adjusted using a minimum loss function. Upon completion of training, a preset fourth emotion recognition model is obtained. The fourth emotion recognition model is used to perform emotion recognition processing on at least two of the speech data, the text data, and the image data.

[0204] Optionally, the process executed by processor 301 to perform emotion recognition processing on the voice data and / or image data using a preset emotion recognition model to obtain the emotion recognition result of the target user includes:

[0205] The voice data is processed by a preset first emotion recognition model to obtain the voice emotion recognition result.

[0206] Alternatively, the voice data can be converted into text data, and the text data can be processed for emotion recognition using a preset second emotion recognition model to obtain the text emotion recognition result.

[0207] Alternatively, the image data can be processed using a pre-defined third emotion recognition model to obtain the image emotion recognition result;

[0208] Alternatively, by using a preset fourth emotion recognition model, emotion recognition processing can be performed on at least two of the voice data, text data, and image data to obtain multimodal emotion recognition results;

[0209] Based on the voice emotion recognition result, or the text emotion recognition result, or the image emotion recognition result, or the multimodal emotion recognition result, the emotion recognition result of the target user is obtained.

[0210] Optionally, before processing the user's question-and-answer data based on the emotion recognition results, combined with long short-term memory knowledge and the target knowledge base, to obtain answer data, the processor 301 further performs the following:

[0211] Acquire the user's short-term memory data, which includes the user's historical dialogue records, user intent, and user emotional state data within a first preset time period;

[0212] Based on the aforementioned short-term memory data, short-term memory knowledge is constructed;

[0213] Acquire the user's long-term memory data, which includes the user's profile data and event data within a second preset time period, the second preset time period being longer than the first preset time period;

[0214] Based on the aforementioned long-term memory data, long-term memory knowledge is constructed.

[0215] Optionally, the processor 301 performs question-and-answer processing on the user's question-and-answer data based on the emotion recognition result, combined with long short-term memory knowledge and the target knowledge base, to obtain answer data, including:

[0216] Based on the emotion recognition results and long short-term memory knowledge, a corresponding emotion processing strategy is generated in the target knowledge base;

[0217] Based on the aforementioned emotion processing strategy, the question-and-answer data is processed to obtain answer data.

[0218] Optionally, the processor 301's execution of the action of controlling the smart toy to output the answer data includes:

[0219] Based on the semantics of the answer data, the output mode of the smart toy is determined to be voice mode and / or action mode;

[0220] Based on the voice pattern and / or action pattern, the smart toy is controlled to output the answer data via voice and / or action.

[0221] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the intelligent toy control method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0222] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0223] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A control method for an intelligent toy, characterized in that, The method includes the following steps: When the smart toy is in a question-and-answer state, it acquires the user's voice data and / or image data; The user's emotion recognition result is determined by performing emotion recognition processing on the voice data and / or image data using a preset emotion recognition model. Based on the emotion recognition results, combined with long short-term memory knowledge and the target knowledge base, the user's question and answer data is processed to obtain answer data. The level of the target knowledge base corresponds to the current growth value of the smart toy. Different levels of the target knowledge base have knowledge data corresponding to different user ages. Control the smart toy to output the answer data.

2. The method as described in claim 1, characterized in that, Before acquiring the user's voice data and / or image data when the smart toy is in a question-and-answer state, the method further includes: When the smart toy is activated, it acquires the user's initial voice data. Based on the first voice data, determine whether the smart toy is in a question-and-answer state; If the smart toy is in a question-and-answer state, then it acquires second voice data and / or image data.

3. The control method for the intelligent toy as described in claim 2, characterized in that, Before performing emotion recognition processing on the voice data and / or image data using a preset emotion recognition model to determine the user's emotion recognition result, the method further includes: Obtain a speech training dataset and a pre-trained emotion recognition model. The training dataset includes sample speech data and emotion annotation data corresponding to the speech data. The pre-trained emotion recognition model is trained using the speech training dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using a minimum loss function. Once the training is complete, a preset first emotion recognition model is obtained. The first emotion recognition model is used to perform emotion recognition processing on the speech data. Alternatively, obtain a text dataset and a pre-trained emotion recognition model, wherein the text dataset is obtained by speech-to-text processing based on a speech training dataset; The pre-trained emotion recognition model is trained using the text dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using the minimum loss function. After training is completed, a preset second emotion recognition model is obtained. The second emotion recognition model is used to perform emotion recognition processing on the text data obtained after the speech data has been processed into text. Alternatively, obtain an image training dataset and a pre-trained emotion recognition model, wherein the image training dataset includes sample image data and emotion annotation data corresponding to the sample image data; The pre-trained emotion recognition model is trained using the image training dataset. During the training process, the parameters of the pre-trained emotion recognition model are adjusted using the minimum loss function. After training is completed, a preset third emotion recognition model is obtained. The third emotion recognition model is used to perform emotion recognition processing on the image data. Alternatively, obtain at least two of the following: a speech training dataset, a text training dataset, and an image training dataset, as well as a pre-trained emotion recognition model. The speech training dataset includes sample speech data and emotion annotation data corresponding to the sample speech data. The image training dataset includes sample image data and emotion annotation data corresponding to the sample image data. The pre-trained emotion recognition model is trained using at least two of the speech training dataset, the text dataset, and the image training dataset. During training, the parameters of the pre-trained emotion recognition model are adjusted using a minimum loss function. Upon completion of training, a preset fourth emotion recognition model is obtained. The fourth emotion recognition model is used to perform emotion recognition processing on at least two of the speech data, the text data, and the image data.

4. The control method for the intelligent toy as described in claim 3, characterized in that, The step of performing emotion recognition processing on the voice data and / or image data using a preset emotion recognition model to obtain the emotion recognition result of the target user includes: The voice data is processed by a preset first emotion recognition model to obtain the voice emotion recognition result. Alternatively, the voice data can be converted into text data, and the text data can be processed for emotion recognition using a preset second emotion recognition model to obtain the text emotion recognition result. Alternatively, the image data can be processed using a pre-defined third emotion recognition model to obtain the image emotion recognition result; Alternatively, by using a preset fourth emotion recognition model, emotion recognition processing can be performed on at least two of the voice data, text data, and image data to obtain multimodal emotion recognition results; Based on the voice emotion recognition result, or the text emotion recognition result, or the image emotion recognition result, or the multimodal emotion recognition result, the emotion recognition result of the target user is obtained.

5. The control method for the intelligent toy as described in claim 4, characterized in that, Before processing the user's question-and-answer data based on the emotion recognition results, combined with long short-term memory knowledge and the target knowledge base, to obtain the answer data, the process further includes: Acquire the user's short-term memory data, which includes the user's historical dialogue records, user intent, and user emotional state data within a first preset time period; Based on the aforementioned short-term memory data, short-term memory knowledge is constructed; Acquire the user's long-term memory data, which includes the user's profile data and event data within a second preset time period, the second preset time period being longer than the first preset time period; Based on the aforementioned long-term memory data, long-term memory knowledge is constructed.

6. The control method for the intelligent toy as described in claim 5, characterized in that, Based on the emotion recognition results, combined with long short-term memory knowledge and a target knowledge base, the user's question-and-answer data is processed to obtain answer data, including: Based on the emotion recognition results and long short-term memory knowledge, a corresponding emotion processing strategy is generated in the target knowledge base; Based on the aforementioned emotion processing strategy, the question-and-answer data is processed to obtain answer data.

7. The control method for the intelligent toy as described in claim 6, characterized in that, The step of controlling the smart toy to output the answer data includes: Based on the semantics of the answer data, the output mode of the smart toy is determined to be voice mode and / or action mode; Based on the voice pattern and / or action pattern, the smart toy is controlled to output the answer data via voice and / or action.

8. A control device for an intelligent toy, characterized in that, The control device for the intelligent toy includes: The first acquisition module is used to acquire the user's voice data and / or image data when the smart toy is in a question-and-answer state. The first processing module is used to perform emotion recognition processing on the voice data and / or image data through a preset emotion recognition model to determine the emotion recognition result of the user. The second processing module is used to process the user's question-and-answer data based on the emotion recognition results, combined with long short-term memory knowledge and the target knowledge base, to obtain answer data. The level of the target knowledge base corresponds to the current growth value of the smart toy, and different levels of the target knowledge base have knowledge data corresponding to different user ages. The control module is used to control the smart toy to output the answer data.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the control method for the intelligent toy as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the control method for the intelligent toy as described in any one of claims 1 to 7.

Images