Multi-modal data based multi-task intelligent robot

Abstract

The application discloses a kind of multi-task wisdom robots based on multi-modal data, comprising: personalized interaction module obtains scene image and obtains face image by face recognition, and judge whether it is student, if yes, the identity information of student is obtained, and corresponding historical data is obtained;Face expression extraction is carried out based on face image and generates face label and conversation theme in combination with historical behavior data;Conversation content is generated based on face label, conversation theme and historical data, whether student is in speaking state is judged, if yes, conversation is stopped, and new conversation content is generated;Multi-task module pre-set task mode, regional function and schedule, obtain current time and current location, obtain current task mode and collect student dynamic data, generate evaluation report based on multi-task data, at least including student dynamic data and historical data.The efficient and personalized conversation and multi-task service with student are realized by the application.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and more specifically to a multi-task intelligent robot based on multimodal data. Background Technology

[0002] Currently, when intelligent robots communicate and interact with users, the following process is commonly used to achieve natural dialogue: First, automatic speech recognition technology converts the user's input into text; then, LLM (Large Language Model) is used to understand the text content and generate corresponding text responses; finally, text-to-speech technology converts the generated text responses into speech and plays them back to the user, thus completing the interaction. This method has the following problems during the interaction process: 1. It cannot generate personalized interactive content based on student behavioral data, relying entirely on LLM for mechanical responses; 2. It cannot recognize emotional fluctuations, lacking emotional recognition and empathy capabilities, resulting in a rigid approach; 3. It lacks online learning capabilities, relying solely on model improvements and updates, unable to update dialogue based on new knowledge, leading to a decline in user experience after long-term use and a lack of self-optimization capabilities. Furthermore, current smart robots collect only single-mode data and cannot acquire multimodal data. This leads to the inability to record students' daily performance in a timely manner in educational settings, resulting in information omissions or forgetting. This affects the accuracy and timeliness of home-school communication. Relying solely on teachers to provide feedback to each parent requires a significant amount of time and energy. Moreover, personalized evaluation for each child is not only labor-intensive but also affects the rational allocation of teachers' energy. In addition, relying solely on human evaluation makes it difficult to accurately evaluate students, affecting the accuracy of feedback to parents. Furthermore, they cannot undertake multi-task responsibilities such as inspection and teaching assistance, thus limiting their application. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies by providing a multi-task intelligent robot based on multimodal data.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0005] A multi-task intelligent robot based on multimodal data includes a personalized interaction module and a multi-task module;

[0006] The personalized interaction module is set as follows:

[0007] The system acquires scene images and obtains facial images through face recognition. It then determines whether the person in the facial image is a student. If so, it acquires the student's identity information and obtains corresponding historical data, including historical dialogue data and historical behavior data.

[0008] Facial expressions are extracted from facial images and combined with historical behavioral data to generate facial tags and dialogue topics;

[0009] The system generates dialogue content based on facial tags, dialogue topics, and historical data, and determines whether the student is currently speaking. If so, the dialogue stops, and new dialogue content is generated based on the student's current speech. If not, the dialogue continues based on the existing dialogue content.

[0010] The multitasking module is set to:

[0011] The system presets task modes, area functions, and schedules. Based on the current time and location, combined with area functions and schedules, it switches task modes to execute corresponding tasks and collect student dynamic data.

[0012] An evaluation report is generated based on multi-task data, which includes at least student dynamic data and historical data.

[0013] As one possible implementation, the facial expression extraction from the facial image includes the following steps:

[0014] The emotion classification of facial images is performed by an expression classification model to obtain the emotion category. The expression classification model includes EfficientNet.

[0015] The facial expression features corresponding to the student's identity information are formed based on the emotion category.

[0016] As one possible implementation, the facial tags and conversation topics are obtained through the following steps:

[0017] By acquiring historical behavioral data, we obtain event records of teachers interacting with students. Based on these event records, we obtain positive and negative labels for students' emotions and combine them with facial expression features to form facial labels.

[0018] Based on event log data, facial expression features, and facial tags, a retrieval framework is used to retrieve historical behavior data to obtain relevant historical data. Dialogue topics are then generated based on this relevant historical data. The retrieval framework includes the RAG model.

[0019] As one possible implementation, the process of generating dialogue content based on facial tags, dialogue topics, and historical data, and determining whether the student is currently speaking, includes the following steps: If so, the dialogue stops, and new dialogue content is generated based on the student's current speaking content; otherwise, the dialogue continues based on the existing dialogue content.

[0020] The dialogue generation model generates dialogue text based on historical data, facial tags, and dialogue topics. The dialogue text is then converted into dialogue content using a speech model. The dialogue generation model includes an LLM model, and the speech model includes a TTS model.

[0021] The ASR speech recognition method is used to identify and determine whether a student is speaking.

[0022] If the student is currently speaking, the dialogue stops, and new dialogue content is generated based on the student's current speech.

[0023] If not, continue the conversation based on the dialogue content.

[0024] As one possible implementation method, the historical data also includes student work data, based on which students' emotional expression and cognitive development are evaluated.

[0025] As one possible implementation, it also includes periodically judging the student's level of interest in the current conversation, setting an interest threshold, and if the level of interest in the current conversation does not meet the interest threshold, then switching the topic of the current conversation.

[0026] As one possible implementation method, the task mode includes security patrol, classroom data collection, teaching assistance, and attendance tasks;

[0027] The security patrol and classroom data collection obtains the corresponding patrol time and collection time through the schedule plan, and obtains the corresponding patrol location and collection location through the area function. The patrol personnel arrive at the patrol location within the patrol time to carry out the patrol, and report any abnormal situations.

[0028] The classroom data collection is carried out at the collection location within the collection time to collect classroom data and obtain classroom performance data. If there are any abnormalities in the classroom performance data, they are reported.

[0029] The teaching aids obtain corresponding aid time through a schedule plan, obtain teacher evaluation data based on historical data, and generate corresponding dialogue content.

[0030] The attendance task obtains the corresponding attendance time through the schedule plan, obtains the corresponding attendance location through the area function, and obtains the student's identity information through facial recognition. Students arrive at the corresponding attendance location within the attendance time and attendance is recorded based on their identity information.

[0031] As one possible implementation method, the assessment report includes student behavior evaluation, student attendance statistics, student profiles, student tags, and student growth curves.

[0032] This invention, by adopting the above technical solutions, has significant technical effects:

[0033] This invention acquires student facial images and performs identity verification to obtain student identity information. Based on this identity information, it retrieves the student's corresponding historical dialogue and behavioral data. Facial expressions are extracted from the student's facial images to obtain facial feature data, generating emotion tags and dialogue topics. The system determines whether the student is currently speaking; if so, the dialogue stops to prevent interruption. New dialogue content is generated based on the student's speech. Through a schedule, combined with the current time and location, the system identifies the tasks to be performed, collects data for the corresponding tasks, and generates student evaluation reports. This invention's robot, by incorporating private data for dialogue, addresses the lack of personalized dialogue capabilities in current robots, enhancing the enjoyment and interest of student conversations. Furthermore, by collecting student data from multiple angles for multi-faceted analysis and extraction, student information is updated, helping teachers and parents gain a deeper understanding of students and improving the efficiency of home-school communication. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the modules of the system of the present invention;

[0036] Figure 2 This is a schematic diagram of the existing voice interaction process;

[0037] Figure 3 This is a schematic diagram of the personalized interaction process of the present invention;

[0038] Figure 4 This is a schematic diagram of the multi-task execution process of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following embodiments.

[0040] Example 1:

[0041] A multi-task intelligent robot based on multimodal data, such as Figure 1 As shown, it includes a personalized interaction module 100 and a multi-tasking module 200;

[0042] The personalized interaction module is set as follows:

[0043] The system acquires scene images and obtains facial images through face recognition. Based on the facial images, it determines whether the person is a student. If so, it acquires the student's identity information and obtains corresponding historical data through the identity information. The historical data includes historical dialogue data and historical behavior data.

[0044] Facial expressions are extracted from facial images and combined with historical behavioral data to generate facial tags and dialogue topics;

[0045] The system generates dialogue content based on facial tags, dialogue topics, and historical data, and determines whether the student is currently speaking. If so, the dialogue stops, and new dialogue content is generated based on the student's current speech. If not, the dialogue continues based on the existing dialogue content.

[0046] The multitasking module is set to:

[0047] The system presets task modes, area functions, and schedules. Based on the current time and location, combined with area functions and schedules, it switches task modes to execute corresponding tasks and collect student dynamic data.

[0048] An evaluation report is generated based on multi-task data, which includes at least student dynamic data and historical data.

[0049] This embodiment proposes a multi-task intelligent robot based on multimodal data. It combines platform-owned data for dialogue, addressing the current lack of personalized dialogue recommendation capabilities and enhancing students' enjoyment and interest in robot conversations. Through a multi-task module, it collects student data from multiple angles, including dialogue content, voice, video, and images. Multi-angle data extraction and analysis yield corresponding student assessment reports, improving the efficiency of home-school communication. Combined with the robot's navigation capabilities, it patrols using camera capture and AI recognition, promptly capturing students' memorable moments. Simultaneously, it identifies and notifies teachers of risks (such as fires, fights, and smoking), reducing their teaching burden.

[0050] Currently, the industry commonly adopts processes such as... to achieve natural dialogue with users. Figure 2As shown, the process begins with Automatic Speech Recognition (ASR) technology converting the user's voice input into text. ASR utilizes machine learning and deep learning algorithms to convert speech into text, and is widely used in various scenarios such as intelligent assistants, voice interaction in telephone customer service, and voice input methods. It can understand and transcribe human language, enabling computers to "understand" human speech and perform operations or provide services based on voice commands. Subsequently, a Large Language Model (LLM) based on deep learning algorithms (such as Transformer) is used to understand the text content and generate corresponding responses. It can understand and generate natural language and perform various tasks such as translation, question answering, summary generation, and sentence completion. Finally, Text-to-Speech (TTS) technology, the opposite of ASR, is used to convert written text into spoken language. The generated text answer is converted into speech and played back to the user. By using TTS, computers, mobile devices, or other electronic devices can synthesize human speech sounds, enabling machines to interact with people more naturally. It is widely used in customer service automation, navigation systems, and language learning, thus completing the entire voice interaction process.

[0051] Based on existing technologies, current smart robots have the following drawbacks when interacting with students: 1. Lack of personalized recommendation capabilities: They cannot generate personalized content based on student behavior data and rely entirely on the capabilities of the LLM (Limited Learning Model) itself, resulting in a mechanical and rigid dialogue with students; 2. Insufficient emotion recognition and empathy capabilities: Because they do not consider students' emotions, they cannot effectively soothe and understand students when they experience emotional fluctuations through dialogue; 3. Lack of continuous recognition and empathy capabilities: Due to their reliance on the capabilities of the LLM itself, they lack online learning capabilities and can only improve by updating the model. They cannot engage in dialogue with students based on new knowledge, leading to a decline in user experience with long-term use; 4. Functional limitations: Existing smart robots have failed to effectively fulfill responsibilities such as security inspections, daily snapshots, and teaching assistance.

[0052] To address the aforementioned issues and enhance the personalization of the dialogue process, in this embodiment, the intelligent robot incorporates known student information during conversations, such as: current grade level, recent artwork, teacher and parent evaluations, etc. Each dialogue session is uploaded, saving the captured image, audio recording, and text. A schematic diagram of the process is shown below. Figure 3 As shown, this is achieved through a personalized interaction module, specifically including the following steps:

[0053] Step 1: Acquire scene images and obtain facial images through face recognition, and make matching judgments based on facial images and known face databases;

[0054] Step 2: If the matching fails for 10 consecutive frames during the matching process, it means that the current person is an external person. The conversation is conducted through the LLM's own capabilities. If the matching is successful, the student's identity information is obtained, and the student's historical data is retrieved from the database based on the identity information.

[0055] Step 3: Using facial images, perform facial expression classification based on EfficientNet to obtain emotion categories, and form expression features corresponding to the student's identity information based on the emotion categories;

[0056] Step 4: By recording students' historical behavior data by the teacher, positive and negative labels are marked for the students, and facial labels are generated for the students. Based on facial expression features and facial labels, the historical behavior data is retrieved through the RAG model to obtain relevant historical data. The dialogue topic is determined through the relevant historical data.

[0057] Step 5: Using the LLM model, based on historical data, facial tags, and dialogue topics, generate dialogue text. Use the TTS model to convert the dialogue text into dialogue content. Based on ASR speech recognition technology, determine whether the student is speaking. If the student is speaking, stop the dialogue and generate a new dialogue based on the student's speech content; otherwise, continue the dialogue based on the dialogue content.

[0058] The interaction achieved through the method in this embodiment is customized by incorporating the student's private data, rather than relying entirely on the LLM's own dialogue capabilities. Facial recognition technology is used to control the dialogue process, ensuring that the content of the interactive dialogue matches the user. Current smart robots often use "start" and "pause" buttons to interrupt and control the chat process. The method in this embodiment continuously monitors the student's voice input; once a student is detected speaking, the currently unfinished dialogue is automatically interrupted, and a new dialogue is started, achieving a natural, immediate, and interruptible interactive experience.

[0059] This embodiment also includes: if the student's historical data contains artwork data, such as paintings, the artwork data is analyzed and extracted to obtain the artwork content. The artwork content is used to identify the student's emotional expression, and the artwork content and emotional expression are fed back to help the robot generate dialogue topics that the student is likely to be interested in; a preset time threshold is set, and if the student does not initiate a dialogue within the time threshold, dialogue content is automatically generated; the student's level of interest in the current dialogue is periodically detected, and if the detection result shows that the student's level of interest in the current dialogue is low, the current dialogue topic is switched. For example, after every 5 rounds of dialogue, the dialogue content is recorded and the student's level of interest is calculated. If the level of interest shows that the student is not interested in the current topic, a new dialogue topic is determined based on historical data.

[0060] This embodiment provides a dialogue scenario based on this method. The dialogue topics generated in this embodiment are recommended based on historical dialogue summaries and the expert knowledge base of the preschool guide, as shown below:

[0061] Scene 1:

[0062] A child walks up to the robot. Facial recognition identifies the child as "Child A" from the middle class. By accessing Child A's historical dialogue and behavioral data, including recent data from camera, voice recognition, and teacher recordings, the robot analyzes Child A's emotional fluctuations. It discovers that Child A's favorite red block was taken away while building with blocks; during morning exercise, Child A kept their head down and remained silent; and Child A was not active during games. Based on this data, a gentle dialogue strategy is generated as follows: "Hi, Child A! Welcome back! I just noticed your 'mood star' is a little dim today. Were you a little sad after your nap?"

[0063] And conduct the following dialogue interaction:

[0064] The robot said, "I know that sometimes moods can drift by like dark clouds, but that's okay. How about we draw a 'mood rainbow' together? We can blow it up into the sky and chase away the dark clouds!"

[0065] The robot's screen displays a mechanical arm extending out and handing over a "magic paintbrush." ​​A rainbow palette then appears on the paintbrush, followed by the following dialogue:

[0066] "Come on, little one, let's draw a rainbow that's just for you. Each color represents a thought in your heart."

[0067] Child A started drawing, saying as he drew, "This yellow color is from the orange candy I ate yesterday, I like it."

[0068] The robot responded, "Wow, then you can give this candy to that little kid from yesterday, and it can become a bridge of friendship!"

[0069] Scene 2:

[0070] A child stands crying in front of the robot. Facial recognition identifies the child as Child B. The robot retrieves Child B's historical data and identifies a recent drawing of a picnic scene. Analysis of the drawing reveals a blue sky reflected in clear river water, with many children playing – some flying kites, some playing drop the handkerchief, and some having a water fight. The interaction process based on the drawing's content is as follows:

[0071] Robot: "Little friend B, your painting has a river, big trees, and lots of children playing. Can you tell me what your favorite game is? Is it flying a kite, dropping the handkerchief, or having a water fight?"

[0072] Child B: "I love water fights the most."

[0073] Based on the child's dialogue, we can tell that the child is interested in the current drawing. Therefore, we will continue the interaction based on the drawing until the child loses interest.

[0074] Scene 3:

[0075] The system uses facial recognition to identify students, retrieves historical data based on that information, and extracts key events, such as photos taken during morning access control. It then uses multimodal analysis to analyze students' emotions upon arrival and generates relevant dialogues, such as: "Little friend C, I saw you crying and looking unhappy this morning. Did something upset you?"

[0076] For example, after confirming the student's identity information, the teacher's event records for the child can be retrieved through RAG, summarized, and a summary can be generated to produce a dialogue topic such as: "Little friend C, I heard that you participated in the Little Seagull Star election today, and you were the first to raise your hand?"

[0077] For example, record five rounds of dialogue data to identify interest levels and determine whether to switch topics. Generate the following dialogue: "Little friend C, you don't seem very interested in today's events. My super data shows that your artwork from the day before yesterday was very vivid, and it had a cute big tabby cat in it. Can you tell me about this big tabby cat?"

[0078] If the student remains in front of the robot without speaking, the robot can initiate a topic, such as, "I can tell stories, let's play some riddles together!"

[0079] This embodiment utilizes a multi-task module, as illustrated in the flowchart below. Figure 4As shown, the robot has preset task modes, area functions, and schedules. Based on a combination of time perception and location recognition, and according to the learning schedule and its own positioning, it can realize multi-task intelligent scheduling and automatic task execution on campus, dynamically switching to the corresponding task mode. These tasks include security patrol, classroom data collection, teaching assistance, and attendance tasks. It can also capture images during meals, lull children to sleep during nap time, and interact with children, achieving "on-time arrival, automatic response, and task closure." Furthermore, it is responsible for collecting dynamic student data, including images of students' daily behavior, video recording of highlight moments, reporting of abnormal behavior in activity areas, and casual conversations about daily activities. Specifically, the following steps are included:

[0080] Step 1: Set up the task mode, area functions, and schedule to prepare for task execution;

[0081] Step 2: Obtain the robot's current time and location in real time through environmental perception and location recognition, obtain the corresponding current task and area function based on the current time, and automatically match and switch to the current task mode;

[0082] Step 3: Task execution. Execute tasks according to the current task mode, completing the corresponding designated tasks in different time periods and areas, such as facial recognition attendance, anomaly detection, in-park snapshots, playing soothing music, and children's interaction.

[0083] Step 4: After the task is completed, the robot will autonomously navigate to the next task point according to the schedule, task mode and area function to continue to perform the task.

[0084] The following is an example of the robot's multi-tasking capabilities in this embodiment:

[0085] Scene 1:

[0086] Security patrols take place daily from 4:00 PM to 5:00 PM. The robot patrols the designated area and takes photos at each patrol point. If it detects fire or dangerous items such as knives, it will notify the relevant school administrators. After completing the task, the robot uploads the photos to the database for storage. From 8:00 PM to 9:00 PM daily, the robot begins nighttime security patrols. Upon reaching the designated patrol point, it performs facial recognition. If a face is detected, it takes a photo and notifies the school administrators.

[0087] Scene 2:

[0088] Classroom data collection: Between 8:00 and 9:00, the robot uses task modes and area functions to obtain information on the locations where classroom data collection will be performed, collects students' classroom performance data, and performs temperature checks through facial recognition. If a child with an abnormal temperature is found, the relevant supervising teacher will be notified. During the classroom data collection process, students' classroom performance data, including classroom images, classroom videos, and classroom dialogues, will be collected.

[0089] Scene 3:

[0090] The attendance task involves arriving at the designated attendance point within the specified time and undergoing facial recognition. If the child is successfully identified as "Little Ding," a voice will announce, "Welcome, Little Ding!" When Little Ding is identified at the end of the afternoon, a voice will announce, "See you tomorrow, Little Ding!" After the attendance task is completed, the captured photo and conversation data will be uploaded to the database.

[0091] In this embodiment, the method also includes uploading historical data from the robot interaction process and dynamic data of students during task execution to a database for storage. By periodically extracting and summarizing student information, student evaluation reports are obtained, such as daily, weekly, and monthly summaries. After being reviewed by the teacher, the reports are sent to the parents in the form of reports, allowing parents to better understand their child's performance at school. This also helps teachers to identify students' shortcomings in a timely manner and provide targeted guidance. Based on the method of this embodiment, a comprehensive evaluation of students can be achieved.

[0092] Various changes and modifications made without departing from the spirit and scope of this invention, and all equivalent technical solutions, also fall within the scope of this invention.

[0093] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0094] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] This invention is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0098] It should be noted that:

[0099] The phrase "an embodiment" or "an embodiment" used in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Therefore, the phrase "an embodiment" or "an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0100] Furthermore, it should be noted that the shapes and names of the parts and components described in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, as long as they do not depart from the structure of this invention or exceed the scope defined in these claims, they should all fall within the protection scope of this invention.

Claims

1. A multi-task intelligent robot based on multimodal data, characterized in that, Includes personalized interaction modules and multi-tasking modules; The personalized interaction module is set as follows: The system acquires scene images and obtains facial images through face recognition. Based on the facial images, it determines whether the person is a student. If so, it acquires the student's identity information and obtains corresponding historical data through the identity information. The historical data includes historical dialogue data and historical behavior data. Facial expressions are extracted from facial images and combined with historical behavioral data to generate facial tags and dialogue topics; The step of extracting facial expressions from a facial image includes the following steps: The emotion classification of facial images is performed by an expression classification model to obtain the emotion category. The expression classification model includes EfficientNet. Based on emotion categories, facial expression features are generated corresponding to students' identity information; The facial tags and conversation topics are obtained through the following steps: By acquiring historical behavioral data, we obtain event records of teachers interacting with students. Based on these event records, we obtain positive and negative labels for students' emotions and combine them with facial expression features to form facial labels. Based on event log data, facial expression features, and facial tags, a retrieval framework is used to retrieve historical behavior data to obtain relevant historical data. Dialogue topics are then generated based on the relevant historical data. The retrieval framework includes the RAG model. The system generates dialogue content based on facial tags, dialogue topics, and historical data, and determines whether the student is currently speaking. If so, the dialogue stops, and new dialogue content is generated based on the student's current speech. If not, the dialogue continues based on the existing dialogue content. The multitasking module is set to: The system presets task modes, area functions, and schedules. Based on the current time and location, combined with the area functions and schedules, it switches task modes to execute corresponding tasks and collect student dynamic data. An evaluation report is generated based on multi-task data, which includes at least student dynamic data and historical data.

2. The multi-task intelligent robot based on multimodal data according to claim 1, characterized in that, The process of generating dialogue content based on facial tags, dialogue topics, and historical data, and determining whether the student is currently speaking, includes the following steps: If so, the dialogue stops, and new dialogue content is generated based on the student's current speech; otherwise, the dialogue continues based on the existing dialogue content. The dialogue generation model generates dialogue text based on historical data, facial tags, and dialogue topics. The dialogue text is then converted into dialogue content using a speech model. The dialogue generation model includes an LLM model, and the speech model includes a TTS model. The ASR speech recognition method is used to identify and determine whether a student is speaking. If the student is currently speaking, the dialogue stops, and new dialogue content is generated based on the student's current speech. If not, continue the conversation based on the dialogue content.

3. The multi-task intelligent robot based on multimodal data according to claim 1, characterized in that, The historical data also includes student work data, which is used to assess students' emotional expression and cognitive development.

4. The multi-task intelligent robot based on multimodal data according to claim 1, characterized in that, It also includes periodically judging the student's level of interest in the current conversation, setting an interest threshold, and if the level of interest in the current conversation does not meet the interest threshold, then switching the topic of the current conversation.

5. The multi-task intelligent robot based on multimodal data according to claim 1, characterized in that, The task modes include security patrol, classroom data collection, teaching assistance, and attendance tasks; The security patrol and classroom data collection obtains the corresponding patrol time and collection time through the schedule plan, and obtains the corresponding patrol location and collection location through the area function. The patrol personnel arrive at the patrol location within the patrol time to carry out the patrol, and report any abnormal situations. The classroom data collection is carried out at the collection location within the collection time to collect classroom data and obtain classroom performance data. If there are any abnormalities in the classroom performance data, they are reported. The teaching aids obtain corresponding aid time through a schedule plan, obtain teacher evaluation data based on historical data, and generate corresponding dialogue content. The attendance task obtains the corresponding attendance time through the schedule plan, obtains the corresponding attendance location through the area function, and obtains the student's identity information through facial recognition. Students arrive at the corresponding attendance location within the attendance time and attendance is recorded based on their identity information.

6. The multi-task intelligent robot based on multimodal data according to claim 1, characterized in that, The assessment report includes student behavior evaluation, student attendance statistics, student profiles, student tags, and student growth curves.

Images