A multi-modal interaction learning and interaction method and system for a child education robot
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN NADRAY INNOVATIONS TECH CO LTD
- Filing Date
- 2026-04-10
- Publication Date
- 2026-06-12
Smart Images
Figure CN122199229A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of interactive learning technology, and more specifically, to a multimodal interactive learning method and system for children's educational robots. Background Technology
[0002] The existing one-way, didactic learning model is difficult to meet the needs of personalized and highly participatory modern education. From early computer-assisted instruction to today's virtual reality (VR), augmented reality (AR), and intelligent dialogue systems, interactive technologies are constantly breaking the limitations of time and space, enabling learners to engage in two-way dynamic communication with learning content through various means such as touch, voice, gestures, and even eye tracking.
[0003] Current interactive learning primarily involves collecting learners' multimodal input through a front-end interaction layer (such as touchscreens, microphones, and cameras). The back-end logic layer uses an event-driven model to parse user intent and calls knowledge bases or algorithm models to generate targeted content. Finally, the results are presented to learners in the form of text, images, or voice through an instant feedback mechanism to complete the learning interaction. However, in multimodal interactive learning with children's educational robots, when processing multimodal interactive information, the robots typically use early fusion or direct splicing to generate a single intent assumption. This ignores the semantic inconsistencies between different modal information over time and the inherent ambiguity of children as immature interactive subjects when encoding information. This simple fusion strategy makes it easy for children's educational robots to misjudge local modal features as global intents, causing interaction biases during children's interactive learning. Therefore, how to achieve effective interactive learning between children's educational robots and children under the influence of semantic inconsistencies between different modal information has become a challenge for the industry. Summary of the Invention
[0004] This application provides a multimodal interactive learning method and system for children's educational robots, which can realize effective interactive learning between children's educational robots and children under the influence of semantic inconsistency of different modal information.
[0005] In a first aspect, this application provides a multimodal interactive learning method for children's educational robots, comprising the following steps: In response to the first modal interaction information emitted by the target child in the current interactive learning task, an initial intent hypothesis is generated, which is used to indicate the first prediction result of the child education robot on the learning needs of the target child; In response to the second modal interaction information emitted by the target child in the current interactive learning task, the degree of matching between the second modal interaction information and the initial intention hypothesis is calculated; If the matching degree is greater than or equal to a preset threshold, then the intention of generating the first modal interaction information and the second modal interaction information is generated. Figure 1 To send a signal, and based on the stated intent Figure 1 The signal controls the children's educational robot to perform corresponding positive learning guidance actions; If the matching degree is less than the preset threshold, then clarification guidance information is generated based on the difference between the initial intent assumption and the second modal interaction information, and the child education robot is controlled to output the clarification guidance information to guide the target child to make interaction corrections.
[0006] In some embodiments, the method further includes: in response to the current interactive learning task, the children's educational robot collects first modal interaction information of the target child through an audio sensor, wherein the first modal interaction information is a voice interaction stream in the current interactive learning.
[0007] In some embodiments, generating an initial intent hypothesis in response to first modal interaction information emitted by the target child in the current interactive learning task specifically includes: Acquire the first modal interaction information emitted by the target child in the current interactive learning task, and segment at least one voice instruction fragment from the first modal interaction information; The speech instruction segment is subjected to speech recognition to obtain the corresponding text instruction, the text instruction is subjected to intent parsing to obtain a speech intent label, and the intent label is used as the initial intent hypothesis.
[0008] In some embodiments, the method further includes: in response to the current interactive learning task, the children's educational robot collects second modal interactive information of the target child through a visual sensor, wherein the second modal interactive information is a sequence of limb movements of the target child in the current interactive learning.
[0009] In some embodiments, in response to second modal interaction information emitted by the target child in the current interactive learning task, calculating the degree of matching between the second modal interaction information and the initial intent hypothesis specifically includes: The initial intent assumption is mapped to a preset intent encoding space to obtain the first intent encoding, and the action intent label in the second modal interaction information is mapped to the same intent encoding space to obtain the second intent encoding; Calculate the semantic similarity between the first intent encoding and the second intent encoding, and use the semantic similarity as the matching degree between the second modal interaction information and the initial intent hypothesis.
[0010] In some embodiments, if the matching degree is greater than or equal to a preset threshold, then the intention of generating the first modal interaction information and the second modal interaction information is generated. Figure 1 The signals specifically include: When the matching degree is greater than or equal to the preset threshold, it is determined that the first modal interaction information and the second modal interaction information point to the same learning target, and an intention is generated. Figure 1 Send a signal; According to the meaning Figure 1 The signal is sent to align and fuse the voice interaction stream in the first modal interaction information with the body movement sequence in the second modal interaction information to generate a learning intent vector; The corresponding positive learning guidance action sequence is obtained based on the learning intention vector, and the voice output unit of the child education robot is controlled to play the voice guidance content in the positive learning guidance action sequence. At the same time, the action output unit of the child education robot is controlled to execute the physical guidance actions in the positive learning guidance action sequence.
[0011] In some embodiments, if the matching degree is less than the preset threshold, generating clarification guidance information based on the difference between the initial intent assumption and the second modal interaction information specifically includes: Obtain the language intent label corresponding to the initial intent hypothesis and the action intent label corresponding to the second modal interaction information, and calculate the difference vector between the language intent label and the action intent label; Based on the difference vector, a corresponding clarification question frame is obtained by matching it in a preset clarification template library. The language intent tag and the action intent tag are then filled into the clarification question frame to generate the clarification guidance information.
[0012] Secondly, this application provides a multimodal interactive learning system for children's educational robots, used to execute a multimodal interactive learning method for children's educational robots, the system comprising: The response module is used to generate an initial intent hypothesis in response to the first modal interaction information emitted by the target child in the current interactive learning task. The initial intent hypothesis is used to indicate the first prediction result of the child education robot on the learning needs of the target child. The response module is configured to respond to the second modal interaction information emitted by the target child in the current interactive learning task, and calculate the matching degree between the second modal interaction information and the initial intention hypothesis; The execution module is further configured to, if the matching degree is greater than or equal to a preset threshold, generate the intention of the first modal interaction information and the second modal interaction information. Figure 1 To send a signal, and based on the stated intent Figure 1 The signal controls the children's educational robot to perform corresponding positive learning guidance actions; The execution module is further configured to, if the matching degree is less than the preset threshold, generate clarification guidance information based on the difference between the initial intent assumption and the second modal interaction information, and control the children's educational robot to output the clarification guidance information to guide the target child to make interactive corrections.
[0013] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the above-described multimodal interactive learning method for children's educational robots.
[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned multimodal interactive learning method for children's educational robots.
[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The multimodal interactive learning method and system for children's educational robots provided in this application firstly generates an initial intention hypothesis in response to the first modal interactive information emitted by the target child in the current interactive learning task. This initial intention hypothesis is used to indicate a first prediction result of the children's educational robot regarding the target child's learning needs. Secondly, in response to the second modal interactive information emitted by the target child in the current interactive learning task, a matching degree between the second modal interactive information and the initial intention hypothesis is calculated. Then, if the matching degree is greater than or equal to a preset threshold, an intention hypothesis is generated between the first modal interactive information and the second modal interactive information. Figure 1 To send a signal, and based on the stated intent Figure 1 The signal controls the children's educational robot to perform corresponding positive learning guidance actions; finally, if the matching degree is less than the preset threshold, clarification guidance information is generated based on the difference between the initial intention assumption and the second modal interaction information, and the children's educational robot is controlled to output the clarification guidance information to guide the target child to make interaction corrections.
[0016] Therefore, this application demonstrates that it can achieve effective interactive learning between children's educational robots and children under the influence of semantic inconsistencies in different modal information. Firstly, by generating an initial intent hypothesis in response to the first modal interaction information, a verifiable intent benchmark can be established after the child issues a voice command, transforming the originally ambiguous speech stream into a structured intent label. This provides a clear reference for subsequent cross-modal comparisons, avoiding computational redundancy and error accumulation caused by complex fusion of multimodal data. Secondly, by calculating the matching degree in response to the second modal interaction information, a quantitative assessment of the semantic consistency between the child's voice intent and body movement intent is achieved. This provides an adjustable branching basis for the subsequent decision-making of the children's educational robot, solving the technical deficiency of single-modal recognition being unable to determine intent conflicts due to missing information. Then, when the matching degree is greater than or equal to a preset threshold, an intent is generated... Figure 1 By sending signals and executing positive learning guidance actions, the system can promptly trigger teaching feedback that precisely matches the learning objective after confirming that the child's multimodal expressions are directed towards the same learning objective. This avoids outputting irrelevant or incorrect content due to misjudgment of intent, significantly improving the real-time accuracy of interaction and the relevance of teaching content. Finally, when the matching degree is less than a preset threshold, clarifying guidance information is generated and output based on the difference between the initial intent assumption and the second modal interaction information. This accurately locates the conflict dimension, thereby avoiding semantic inconsistencies in the time series of different modal information and interaction deviations caused by the inherent ambiguity of children as immature interaction subjects when encoding information. This enhances the robustness of the children's educational robot in real-world scenarios. In summary, the technical solution provided in this application can achieve effective interactive learning between children's educational robots and children under the influence of semantic inconsistencies in different modal information. Attached Figure Description
[0017] Figure 1 This is an exemplary flowchart of a multimodal interactive learning method for children's educational robots according to some embodiments of this application; Figure 2 This is an exemplary flowchart illustrating the determination of matching degree according to some embodiments of this application; Figure 3 This is a schematic diagram of the structure of a multimodal interactive learning system for children's educational robots, as shown in some embodiments of this application; Figure 4 This is a schematic diagram of the structure of a computer device for implementing a multimodal interactive learning method for children's educational robots, according to some embodiments of this application. Detailed Implementation
[0018] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.
[0019] In the embodiments of this application, at least one can also be described as one or more, and multiple can be two, three, four or more, and this application does not impose any restrictions.
[0020] To better understand the technical solution of this application, the technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0021] refer to Figure 1 The figure is an exemplary flowchart of a multimodal interactive learning method for children's educational robots according to some embodiments of this application. The figure mainly includes the following steps: In step S101, in response to the first modal interaction information emitted by the target child in the current interactive learning task, an initial intent hypothesis is generated, which is used to indicate the first prediction result of the child education robot on the learning needs of the target child.
[0022] Specifically, in response to the current interactive learning task, the children's educational robot collects the first modal interaction information of the target child through an audio sensor. The first modal interaction information is the voice interaction stream in the current interactive learning, which contains at least one voice command segment. Children often exhibit the phenomenon of "saying and doing different things" in the interaction with the educational robot. For example, they may say "red block" but point to the blue block. Relying on a single modality alone cannot accurately identify their true learning intention. By collecting data from these multiple modalities simultaneously, a comparative benchmark can be provided for subsequent intention conflict detection, thereby determining whether the child's verbal expression and physical operation point to the same learning goal. This avoids the robot giving incorrect feedback due to misreading, improving the accuracy of the interaction and the learning effect.
[0023] It should be noted that the first modal interaction information in this application refers to the voice interaction stream of the target child in the current interactive learning task, which is collected by an audio sensor and used to make a preliminary judgment on the child's learning needs in verbal expression.
[0024] In some embodiments, generating an initial intent hypothesis in response to a first modal interaction message emitted by the target child in the current interactive learning task is achieved through the following steps: Acquire the first modal interaction information emitted by the target child in the current interactive learning task, and segment at least one voice instruction fragment from the first modal interaction information; The speech instruction segment is subjected to speech recognition to obtain the corresponding text instruction, the text instruction is subjected to intent parsing to obtain a speech intent label, and the intent label is used as the initial intent hypothesis.
[0025] In specific implementation, firstly, the first modal interaction information emitted by the target child in the current interactive learning task is acquired. This first modal interaction information is the voice interaction stream in the current interactive learning. Voice endpoint detection is performed on the voice interaction stream, that is, the voice interaction stream is divided into multiple short-time analysis frames using a frame-segmentation method. The short-time energy and zero-crossing rate of each short-time analysis frame are calculated. Short-time analysis frames with short-time energy higher than a preset threshold and zero-crossing rate within the typical range of voice are identified as voice segments. Then, at least one voice instruction segment is segmented from the voice interaction stream. The voice instruction segment is a voice unit with temporal semantic integrity in the current interactive learning. Taking the target child saying "I want to learn addition" as an example, after the robot collects the continuous voice stream containing this sentence, it segments the voice instruction segment from the pronunciation of "I" to the pronunciation of "addition" using the endpoint detection method described above. Then, the voice instruction segment is input into a system based on a Hidden Markov Model. The constructed acoustic recognizer extracts the Mel-frequency cepstral coefficient feature vector sequence of the speech command segment and uses the Viterbi algorithm to perform state decoding in a pre-trained acoustic model and language model to output the corresponding text command, such as "I want red blocks". Finally, the text command is subjected to dictionary-based intent parsing, which involves segmenting the text command to obtain a word sequence and matching it with trigger words in a preset intent rule base. For example, "red" matches the color attribute and "blocks" matches the object category. This extracts the speech intent label representing the learning needs of the target child, such as "select red blocks". This intent label is used as the initial intent hypothesis for comparison with the second modality information. The speech intent label refers to the semantic category of the learning goal or operation instruction to be expressed in the speech interaction flow, such as "select red blocks", "assemble triangles", "move forward", etc., which are used to represent the cognitive intent conveyed by the child through spoken language.
[0026] It should be noted that the initial intent hypothesis in this application refers to the first prediction result of the target child's current learning needs generated by the children's educational robot based on the first modal interaction information emitted by the target child. Since children often have the problem of "saying and doing not matching" in their interactions, a single modality cannot reliably lock the true intent. Therefore, voice information is used to quickly form a verifiable initial hypothesis as a benchmark for subsequent comparisons to determine whether the child's multimodal expressions point to the same learning goal, and to avoid the robot giving incorrect feedback due to misreading a single modality.
[0027] In step S102, in response to the second modal interaction information emitted by the target child in the current interactive learning task, the matching degree between the second modal interaction information and the initial intention hypothesis is calculated.
[0028] Specifically, in response to the current interactive learning task, the children's educational robot collects the second modal interactive information of the target child through a visual sensor. The second modal interactive information is the sequence of the target child's limb movements in the current interactive learning. The sequence of limb movements includes at least one gesture trajectory segment. At least one gesture trajectory segment is extracted from the sequence of limb movements, and the corresponding action intention label is obtained by trajectory recognition of the gesture trajectory segment.
[0029] It should be noted that the second modal interactive information in this application refers to the sequence of limb movements of the target child in the current interactive learning task, which is collected by a visual sensor and is used to verify whether the child's verbal expression and limb operation point to the same learning goal.
[0030] In some embodiments, extracting at least one gesture trajectory segment from the limb movement sequence and performing trajectory recognition on the gesture trajectory segment to obtain the corresponding action intention label is achieved through the following steps: The limb movement sequence is obtained, and hand key points are detected in each frame of the limb movement sequence to obtain a hand coordinate sequence; The hand coordinate sequence is temporally segmented to obtain at least one gesture trajectory segment, and the gesture trajectory segment is input into a pre-trained trajectory classifier to output the corresponding action intention label.
[0031] In specific implementation, firstly, the limb movement sequence is acquired. This sequence is a collection of multiple frames of images continuously acquired by the visual sensor of the children's educational robot, containing images of the target child's hand movements. For example, 30 RGB images captured from the moment the child begins to raise their hand until they complete the pointing action. For each frame in the limb movement sequence, hand keypoint detection based on a convolutional neural network is performed. Specifically, an encoder-decoder structured pose estimation network can be used. This pose estimation network outputs the two-dimensional coordinates of the wrist, each phalanx, and fingertip in each frame through pixel-by-pixel regression, thus forming a dynamic hand coordinate sequence covering the entire duration of the action. Each frame corresponds to a set of coordinate points, and the coordinate points of all frames are arranged in chronological order. The hand coordinate sequence is then used; subsequently, a temporal segmentation method based on velocity and acceleration thresholds is employed to detect the start and end frames of hand movement in the hand coordinate sequence, thereby segmenting at least one independent gesture trajectory segment, such as a coordinate subsequence from a static hand position to a static hand position. The gesture trajectory segment is then input into a pre-trained trajectory classifier, which uses a nearest neighbor algorithm based on dynamic time warping to match the input gesture trajectory segment with various gesture templates (such as "point", "swipe", "grab") stored in a template library, calculates the similarity distance of the trajectory shape, selects the template category with the smallest similarity distance as the recognition result, and outputs the corresponding action intention label, such as "pointing at the blue block".
[0032] It should be noted that, in this application, the action intention label refers to the category of the learning object to which the body posture in the sequence of limb movements points, such as pointing a finger at a blue block or drawing a circular trajectory, which is used to characterize the operational intention embodied by the child through limb movements.
[0033] In some embodiments, reference Figure 2 As shown, this figure is an exemplary flowchart of determining the matching degree according to some embodiments of this application. In this embodiment, in response to the second modal interaction information emitted by the target child in the current interactive learning task, the matching degree between the second modal interaction information and the initial intention hypothesis can be calculated by the following steps: In step S1021, the initial intent assumption is mapped to a preset intent encoding space to obtain a first intent encoding, and the action intent label in the second modal interaction information is mapped to the same intent encoding space to obtain a second intent encoding. In step S1022, the semantic similarity between the first intent encoding and the second intent encoding is calculated, and the semantic similarity is used as the matching degree between the second modal interaction information and the initial intent hypothesis.
[0034] In specific implementation, firstly, the initial intent hypothesis is obtained, which is a language intent label obtained based on the parsing of the first modality interaction information, such as "select the red block". Simultaneously, the action intent label from the second modality interaction information is obtained, such as "point to the blue block". Then, the initial intent hypothesis is mapped to a preset intent encoding space to obtain the first intent encoding. This mapping process is achieved by querying a pre-constructed intent encoding mapping table. In this table, each possible language intent label is assigned a fixed-dimensional real-valued vector, which is based on word embedding training on a large amount of children's interactive language corpus. The values obtained through training (e.g., using Word2Vec or BERT models) can reflect the semantic similarity between different intent labels. For example, after training, the intent encoding corresponding to the intent label "select the red block" is a three-dimensional real-valued vector [0.92, 0.15, 0.38]; the intent encoding corresponding to the intent label "select the blue block" is [0.11, 0.87, 0.45]; the intent encoding corresponding to the intent label "select the yellow block" is [0.23, 0.34, 0.91]; and the intent encoding corresponding to the intent label "point to the red block" is [0.89, 0.21, 0.41]. The intent encoding for the intent label "pointing to the blue block" is [0.18, 0.83, 0.52]; the intent encoding for the intent label "grabbing the block" is [0.45, 0.67, 0.22]; and the intent encoding for the intent label "building a house" is [0.76, 0.54, 0.33]. This mapping table is obtained by pre-training with word embeddings on intent labels in children's interactive corpus. Each intent label is fixedly mapped to a unique three-dimensional vector, used to unify the intents parsed from different modalities into the same numerical space. Further details are omitted here. The action intent labels are mapped through the same intent encoding mapping table. The expression is converted into a second intent encoding, for example, mapping "pointing to the blue block" into a vector representation of the same dimension, thereby unifying the expressions of the two different modalities into the same numerical space. Finally, the semantic similarity between the first intent encoding and the second intent encoding is calculated. For example, the cosine similarity calculation method is used, that is, the cosine value of the angle between the two vectors is calculated. The closer the value is to 1, the closer the semantics are. For example, the vector of "selecting the red block" and the vector of "pointing to the blue block" have a low cosine value due to their different color attributes. This semantic similarity is used as the matching degree between the second modal interaction information and the initial intent hypothesis.
[0035] It should be noted that in this application, the matching degree is used to characterize the semantic consistency between the initial intention hypothesis corresponding to the first modal interaction information issued by the target child in the current interactive learning task and the action intention label corresponding to the second modal interaction information. The reason for determining this matching degree is that children generally exhibit the phenomenon of "saying and doing are inconsistent" during the interaction process, and it is impossible to accurately determine their true learning intention by relying on a single modality. Therefore, by quantifying the correlation strength between the two through cross-modal semantic comparison, it is possible to avoid the robot giving incorrect feedback due to misreading, thereby improving the accuracy of interaction and educational effectiveness.
[0036] In step S103, if the matching degree is greater than or equal to a preset threshold, then the intention of the first modal interaction information and the second modal interaction information is generated. Figure 1 To send a signal, and based on the stated intent Figure 1 The signal controls the children's educational robot to perform corresponding positive learning guidance actions.
[0037] In some embodiments, if the matching degree is greater than or equal to a preset threshold, then the intention of generating the first modal interaction information and the second modal interaction information is generated. Figure 1 The signal is achieved through the following steps: When the matching degree is greater than or equal to the preset threshold, it is determined that the first modal interaction information and the second modal interaction information point to the same learning target, and an intention is generated. Figure 1 Send a signal; According to the meaning Figure 1 The signal is sent to align and fuse the voice interaction stream in the first modal interaction information with the body movement sequence in the second modal interaction information to generate a learning intent vector; The corresponding positive learning guidance action sequence is obtained based on the learning intention vector, and the voice output unit of the child education robot is controlled to play the voice guidance content in the positive learning guidance action sequence. At the same time, the action output unit of the child education robot is controlled to execute the physical guidance actions in the positive learning guidance action sequence.
[0038] In specific implementation, firstly, the matching degree is obtained. When the matching degree is greater than or equal to a preset threshold, it is determined that the voice interaction stream corresponding to the first modal interaction information emitted by the target child in the current interactive learning task and the body movement sequence corresponding to the second modal interaction information point to the same learning target. For example, if the child simultaneously says "I want the red block" and points to the location of the red block, a binary form of intention is generated. Figure 1 A signal is sent to trigger the subsequent cross-modal fusion process; secondly, according to the stated intent... Figure 1The signal is sent to the speech interaction stream and the body movement sequence for temporal alignment and fusion. Specifically, each word in the speech interaction stream is first timestamped to obtain the start and end times of the word in the speech interaction stream. At the same time, time labels are attached to the gesture coordinates of each frame in the body movement sequence. Then, an algorithm based on dynamic time warping is used, with the appearance time of the key noun (such as "red building blocks") as the marker, to search for the nearest neighbor gesture pointing frame in the body movement sequence. The intention labels parsed from the speech and the intention labels recognized by the gesture are weighted and merged according to the time correspondence (wherein the weights of the intention labels parsed from the speech and the intention labels recognized by the gesture are not fixed values, but are dynamically adjusted according to the confidence of the current speech recognition and the clarity of the gesture trajectory. As an example, by default, the speech weight and gesture weight can be set to 0.4 and 0.6 respectively, but this weight can be adjusted). Based on real-time needs, an adaptive correction is made (without limitation here), generating a learning intent vector. This vector is a multi-dimensional real-valued vector, with each dimension representing different attributes of the learning target. Finally, based on the learning intent vector, a nearest neighbor search is performed in a preset guidance strategy library to match and obtain a positive learning guidance action sequence corresponding to the learning intent vector. This positive learning guidance action sequence contains at least one voice guidance item and at least one action guidance item. For example, the voice item is "You chose the red block, let's count together," and the action item is the robot nodding and pointing to the red block. The voice output unit of the children's educational robot is controlled to play each sentence in the voice guidance content in sequence, and the action output unit of the children's educational robot is controlled to execute each action in the action guidance according to the time synchronization relationship, thereby realizing multimodal positive guidance output.
[0039] It should be noted that the learning intention vector in this application represents the actual learning needs vector of the target child in the current interactive learning task. Since the intention information of a single modality has the limitation of expression, speech can provide abstract semantic goals but lacks spatial pointing details, and body movements can provide precise spatial positioning but cannot carry a complete semantic description. Cross-modal fusion can complement the perceptual advantages of each and form a more complete description of the child's learning intention.
[0040] In step S104, if the matching degree is less than the preset threshold, then clarification guidance information is generated based on the difference between the initial intention assumption and the second modal interaction information, and the child education robot is controlled to output the clarification guidance information to guide the target child to make interaction corrections.
[0041] In some embodiments, if the matching degree is less than the preset threshold, the generation of clarifying guidance information based on the difference between the initial intent assumption and the second modal interaction information is achieved through the following steps: Obtain the language intent label corresponding to the initial intent hypothesis and the action intent label corresponding to the second modal interaction information, and calculate the difference vector between the language intent label and the action intent label; Based on the difference vector, a corresponding clarification question frame is obtained by matching it in a preset clarification template library. The language intent tag and the action intent tag are then filled into the clarification question frame to generate the clarification guidance information.
[0042] In specific implementation, firstly, the language intent label corresponding to the initial intent hypothesis is obtained, such as "select the red block", and the action intent label corresponding to the second modal interaction information is obtained, such as "point to the blue block". After converting the language intent label and the action intent label into real-valued vectors in the aforementioned intent encoding mapping table, the vector difference between the two is calculated, that is, by subtracting the corresponding dimension of the action intent encoding from each dimension of the first intent encoding to obtain the difference vector. The difference vector is used to quantify the deviation direction and magnitude of the two intents in the semantic space. Subsequently, the difference vector is compared with the distance of each template difference vector in the preset clarification template library. The clarification template library pre-stores clarification question frames corresponding to various common intent conflict types. For example, the clarification question frame corresponding to "color inconsistency" is "Do you want {color A} in the speech or {color B} that the action points to?" This framework uses nearest neighbor matching to select the clarification template with the smallest distance to the current difference vector. It then fills in the attribute words in the language intent label (such as "red") and the attribute words in the action intent label (such as "blue") according to the placeholder positions in the clarification question framework to generate complete clarification guidance information, such as "Do you want the red block or the blue block?".
[0043] It should be noted that the clarification guidance information in this application refers to the specific types of inconsistencies in intent detected by the children's educational robot in the current interaction, as well as the comparative options that need to be clarified by the child. For example, when a child says "red block" but points to "blue block," the question "Do you want the red block or the blue block?" is generated. Since children's "saying and doing are inconsistent" has diverse conflict forms (such as color misalignment, shape misalignment, action type misalignment, etc.), directly asking the child to "say it again" cannot effectively locate the point of conflict. It is necessary to quantify the specific dimensions of the conflict through difference vectors and generate questions containing comparative options in a targeted manner to guide the child to correct themselves with minimal cognitive load.
[0044] In some embodiments, controlling the children's educational robot to output the clarifying guidance information to guide the target child in interactive correction is achieved through the following steps: The clarification guidance information is obtained and input into the speech synthesis unit of the children's educational robot to generate a corresponding clarification speech signal; The speaker of the child education robot is controlled to play the clarification voice signal, and the multimodal acquisition unit of the child education robot is reset to receive the corrective interactive information issued by the target child in response to the clarification guidance information.
[0045] In specific implementation, firstly, the clarification guidance information is obtained. This clarification guidance information is a natural language string, such as "Do you want the red block or the blue block?". This string is input into the speech synthesis unit built into the children's educational robot. This speech synthesis unit uses a synthesis method based on unit selection and waveform concatenation. First, the input string is subjected to text regularization and phonetic conversion to obtain a phoneme sequence. Then, matching primitives are retrieved from a pre-recorded speech segment library based on the phoneme sequence. The optimal primitive sequence is selected by minimizing the concatenation cost function. Finally, the waveform segments corresponding to the primitive sequence are concatenated to generate a continuous clarified speech signal. This clarified speech signal is audio data in pulse code modulation format. Then, the children's educational robot's speaker is controlled to play the clarified speech signal, i.e., the digital-to-analog converter converts the signal into digital data. Audio data is converted into analog electrical signals to drive the speaker diaphragm to vibrate, thereby outputting audible voice questions to the target child. At the end of playback, a reset command is sent to the multimodal acquisition unit of the children's educational robot. This multimodal acquisition unit includes an audio sensor and a visual sensor. The reset command is used to clear the historical audio data in the internal circular buffer of the audio sensor and reset the image acquisition state of the visual sensor to the initial frame. At the same time, it clears the temporary memory occupied by the previously stored first-modal interaction information and second-modal interaction information, so that the acquisition unit enters a ready-to-trigger state to receive the corrective interaction information issued by the target child in response to the clarification guidance information. For example, after hearing the question, the child says "I want the red block" again and points to the red block, thereby providing a brand-new multimodal input for the next round of interaction.
[0046] Furthermore, in another aspect of this application, in some embodiments, this application provides a multimodal interactive learning system for children's educational robots, with reference to... Figure 3 The figure is a schematic diagram of the structure of a multimodal interactive learning system for children's educational robots according to some embodiments of this application. The multimodal interactive learning system for children's educational robots includes a response module 201 and an execution module 202, which are described below: The response module 201 in this application is mainly used to respond to the first modal interaction information emitted by the target child in the current interactive learning task and generate an initial intention hypothesis. The initial intention hypothesis is used to indicate the first prediction result of the child education robot on the learning needs of the target child. The response module 201 is further configured to respond to the second modal interaction information emitted by the target child in the current interactive learning task, and calculate the matching degree between the second modal interaction information and the initial intention hypothesis; Execution module 202, in this application, is mainly used to generate the intention of the first modal interaction information and the second modal interaction information if the matching degree is greater than or equal to a preset threshold. Figure 1 To send a signal, and based on the stated intent Figure 1 The signal controls the children's educational robot to perform corresponding positive learning guidance actions; The execution module 202 is further configured to, if the matching degree is less than the preset threshold, generate clarification guidance information based on the difference between the initial intent assumption and the second modal interaction information, and control the children's educational robot to output the clarification guidance information to guide the target child to make interactive corrections.
[0047] In addition, this application also provides a computer device, the computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the above-described multimodal interactive learning method for children's educational robots.
[0048] In some embodiments, reference Figure 4 The figure is a schematic diagram of the structure of a computer device implementing a multimodal interactive learning method for children's educational robots according to some embodiments of this application. The multimodal interactive learning method for children's educational robots in the above embodiments can be implemented through... Figure 4 The computer device shown is used to implement this, and the computer device includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.
[0049] The processor 301 can be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more devices used to control the execution of the multimodal interactive learning method of the children's educational robot in this application.
[0050] The communication bus 302 can be used to transmit information between the aforementioned components.
[0051] The memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via the communication bus 302. The memory 303 may also be integrated with the processor 301.
[0052] The memory 303 stores program code for executing the scheme of this application, and its execution is controlled by the processor 301. The processor 301 executes the program code stored in the memory 303. The program code may include one or more software modules. In the above embodiments, the determination of the multimodal interactive learning method of the children's educational robot can be achieved by the processor 301 and one or more software modules in the program code in the memory 303.
[0053] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.
[0054] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).
[0055] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.
[0056] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal interactive learning method for children's educational robots.
[0057] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0058] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A multimodal interactive learning method for children's educational robots, characterized in that, Includes the following steps: In response to the first modal interaction information emitted by the target child in the current interactive learning task, an initial intent hypothesis is generated, which is used to indicate the first prediction result of the child education robot on the learning needs of the target child; In response to the second modal interaction information emitted by the target child in the current interactive learning task, the degree of matching between the second modal interaction information and the initial intention hypothesis is calculated; If the matching degree is greater than or equal to a preset threshold, an intent consistency signal is generated between the first modal interaction information and the second modal interaction information, and the child education robot is controlled to perform corresponding positive learning guidance actions based on the intent consistency signal. If the matching degree is less than the preset threshold, then clarification guidance information is generated based on the difference between the initial intent assumption and the second modal interaction information, and the child education robot is controlled to output the clarification guidance information to guide the target child to make interaction corrections.
2. The method as described in claim 1, characterized in that, Also includes: In response to the current interactive learning task, the children's educational robot collects the first modal interaction information of the target child through an audio sensor. The first modal interaction information is the voice interaction stream in the current interactive learning.
3. The method as described in claim 1, characterized in that, In response to the first modal interaction information emitted by the target child in the current interactive learning task, the generation of initial intent hypotheses specifically includes: Acquire the first modal interaction information emitted by the target child in the current interactive learning task, and segment at least one voice instruction fragment from the first modal interaction information; The speech instruction segment is subjected to speech recognition to obtain the corresponding text instruction, the text instruction is subjected to intent parsing to obtain a speech intent label, and the intent label is used as the initial intent hypothesis.
4. The method as described in claim 1, characterized in that, Also includes: In response to the current interactive learning task, the children's educational robot collects the second modal interactive information of the target child through a visual sensor. The second modal interactive information is the sequence of the target child's limb movements during the current interactive learning.
5. The method as described in claim 1, characterized in that, In response to the second modal interaction information emitted by the target child in the current interactive learning task, calculating the matching degree between the second modal interaction information and the initial intention hypothesis specifically includes: The initial intent assumption is mapped to a preset intent encoding space to obtain the first intent encoding, and the action intent label in the second modal interaction information is mapped to the same intent encoding space to obtain the second intent encoding; Calculate the semantic similarity between the first intent encoding and the second intent encoding, and use the semantic similarity as the matching degree between the second modal interaction information and the initial intent hypothesis.
6. The method as described in claim 1, characterized in that, If the matching degree is greater than or equal to a preset threshold, generating an intent consistency signal between the first modal interaction information and the second modal interaction information specifically includes: When the matching degree is greater than or equal to the preset threshold, it is determined that the first modal interaction information and the second modal interaction information point to the same learning target, and an intent consistency signal is generated; Based on the intent consistency signal, the voice interaction stream in the first modal interaction information and the body movement sequence in the second modal interaction information are aligned and fused to generate a learning intent vector; The corresponding positive learning guidance action sequence is obtained based on the learning intention vector, and the voice output unit of the child education robot is controlled to play the voice guidance content in the positive learning guidance action sequence. At the same time, the action output unit of the child education robot is controlled to execute the physical guidance actions in the positive learning guidance action sequence.
7. The method as described in claim 1, characterized in that, If the matching degree is less than the preset threshold, then generating clarification guidance information based on the difference between the initial intent assumption and the second modal interaction information specifically includes: Obtain the language intent label corresponding to the initial intent hypothesis and the action intent label corresponding to the second modal interaction information, and calculate the difference vector between the language intent label and the action intent label; Based on the difference vector, a corresponding clarification question frame is obtained by matching it in a preset clarification template library. The language intent tag and the action intent tag are then filled into the clarification question frame to generate the clarification guidance information.
8. A multimodal interactive learning system for children's educational robots, used to execute the multimodal interactive learning method for children's educational robots as described in any one of claims 1 to 7, characterized in that, The system includes: The response module is used to generate an initial intent hypothesis in response to the first modal interaction information emitted by the target child in the current interactive learning task. The initial intent hypothesis is used to indicate the first prediction result of the child education robot on the learning needs of the target child. The response module is configured to respond to the second modal interaction information emitted by the target child in the current interactive learning task, and calculate the matching degree between the second modal interaction information and the initial intention hypothesis; The execution module is further configured to generate an intent consistency signal between the first modal interaction information and the second modal interaction information if the matching degree is greater than or equal to a preset threshold, and control the children's educational robot to perform corresponding positive learning guidance actions based on the intent consistency signal; The execution module is further configured to, if the matching degree is less than the preset threshold, generate clarification guidance information based on the difference between the initial intent assumption and the second modal interaction information, and control the children's educational robot to output the clarification guidance information to guide the target child to make interactive corrections.
9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing code, and the processor being configured to retrieve the code and execute the multimodal interactive learning method for children's educational robots as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal interactive learning method for children's educational robots as described in any one of claims 1 to 7.