Data processing method and device based on multi-modal intelligent agent

Abstract

The invention provides a data processing method and device based on a multi-modal agent, and is applied to terminal equipment in an intelligent teaching system, and the method comprises the steps: obtaining a target teaching identity of a user and a target teaching scene where the user is located; determining and calling a reference agent corresponding to the target teaching scene as a target agent; collecting user request information input by a user through a target agent, and determining a target response result corresponding to the target application function based on data interaction operation between the target agent and the large model base; and after the target agent receives a target response result returned by the large model base, realizing a target application function according to the target response result. Therefore, the terminal equipment can match the intelligent agent corresponding to the teaching scene according to the user identity, and further provides the corresponding application function service, so that the scene adaptability is improved, and the intelligence and practicability of the teaching system are improved.

Description

Technical Field

[0001] The present application belongs to the general data processing technology field of the Internet industry, and specifically relates to a data processing method and device based on a multimodal intelligent agent. Background Art

[0002] At present, intelligent teaching systems play an important role in the field of education, bringing new conveniences and possibilities to teaching activities and promoting the development of education to a certain extent.

[0003] However, due to the complexity and diversity of teaching scenarios and the different needs of teachers and students, the current intelligent teaching system has obvious technical shortcomings. On the one hand, teaching services are highly homogenized, and it is difficult to provide adaptive functions for the different identities of teachers and students, which makes the teaching equipment less intelligent and unable to accurately meet user needs. On the other hand, the integration and utilization efficiency of teaching resources is low, resulting in poor teaching quality and low efficiency of teaching equipment, which cannot meet the urgent needs of digital transformation of education and restrict the innovative development of education and the improvement of teaching effects. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and device based on a multimodal intelligent agent. The terminal device obtains the user's teaching identity and the corresponding teaching scenario, matches the corresponding reference intelligent agent for the user to interact with, and then provides the user with application functions corresponding to the teaching scenario, thereby improving the scenario adaptability of the terminal device in providing intelligent teaching services, and improving the flexibility and practicality of data processing.

[0005] In a first aspect, an embodiment of the present application provides a data processing method based on a multimodal agent, which is applied to a terminal device in an intelligent teaching system, and the intelligent teaching system also includes a server; the method includes: Obtaining the target teaching identity of the user and the target teaching scenario of the user, where the target teaching identity is used to indicate that the teaching identity corresponding to the user is a teacher identity or a student identity, and the target teaching scenario is any one of multiple reference teaching scenarios matching the teacher identity or the student identity; Determine and call a reference agent corresponding to a target teaching scenario as a target agent, each reference agent corresponds to a reference teaching scenario, the reference agent is used to interact with a user to determine a plurality of preset application functions required to be used in the corresponding reference teaching scenario, and the reference agent is connected to a communication with a large model base pre-set in a server; The target agent collects user request information input by the user, and determines the target response result corresponding to the target application function based on the data interaction operation between the target agent and the large model base. The target application function is determined by the user request information, and the target application function is any one of multiple preset application functions. After the target intelligent agent receives the target response result returned by the large model base, it implements the target application function according to the target response result.

[0006] In a second aspect, an embodiment of the present application provides a data processing device based on a multimodal agent, which is applied to a terminal device in an intelligent teaching system, and the intelligent teaching system also includes a server; the device includes: An information acquisition unit, used to acquire a target teaching identity of a user and a target teaching scenario of the user, wherein the target teaching identity is used to indicate that the teaching identity corresponding to the user is a teacher identity or a student identity, and the target teaching scenario is any one of a plurality of reference teaching scenarios matching the teacher identity or the student identity; An agent calling unit is used to determine and call a reference agent corresponding to a target teaching scene as a target agent, each reference agent corresponds to a reference teaching scene, the reference agent is used to interact with the user to determine a plurality of preset application functions required to be used in the corresponding reference teaching scene, and the reference agent is connected to a communication with a large model base pre-set in the server; A data interaction unit is used to collect user request information input by a user through a target agent, and to determine a target response result corresponding to a target application function based on a data interaction operation between the target agent and the large model base, wherein the target application function is determined by the user request information and is any one of a plurality of preset application functions; The function realization unit is used to realize the target application function according to the target response result after the target intelligent agent receives the target response result returned by the large model base.

[0007] In a third aspect, an embodiment of the present application provides a terminal device, comprising a processor, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the first aspect of the embodiment of the present application.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps in the first aspect of the embodiment of the present application when the computer program / instruction is executed by a processor.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement some or all of the steps described in the first aspect of the embodiment of the present application.

[0010] It can be seen that in the embodiment of the present application, the terminal device can match the intelligent body corresponding to the teaching scene according to the user identity, and then provide the corresponding application function service, improve the scene adaptability to achieve the corresponding intelligent assistance for different links in the teaching process, and improve the intelligence and practicality of the teaching system. The flexibility of the teaching system in executing data processing is improved by matching the intelligent body of different teaching scenes for the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 It is a structural block diagram of an intelligent teaching system provided by an embodiment of the present application; Figure 2 It is a flowchart of a data processing method based on a multimodal agent provided in an embodiment of the present application; Figure 3 is a schematic diagram of a scenario of a triggering event provided in an embodiment of the present application; Figure 4 It is a schematic diagram of the architecture of a target intelligent agent and a large model base provided in an embodiment of the present application; Figure 5 This is a schematic diagram of a teacher teaching scenario provided in an embodiment of the present application; Figure 6 It is a block diagram of the functional units of a data processing device based on a multimodal agent provided in an embodiment of the present application; Figure 7 It is a block diagram of the functional units of another multimodal agent-based data processing device provided in an embodiment of the present application; Figure 8 It is a structural block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0015] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0016] See also Figure 1 , Figure 1 is a structural block diagram of an intelligent teaching system provided in an embodiment of the present application. Figure 1 As shown, the intelligent teaching system 100 includes a terminal device 110 and a server 120. Among them, the terminal device 110 is a common type of electronic device on the market with a display screen that can be used for human-computer interaction by the user 130, including but not limited to computers, smart TVs, mobile phones, etc. The terminal device 110 can interact with the user 130 to start the intelligent teaching service. After detecting that the intelligent teaching service is turned on, the terminal device 110 will first obtain the target teaching identity and target teaching scene of the user 130. The purpose of this operation is to provide a basis for the subsequent selection of reference agents, so that the functions that can be realized by the reference agents meet the user's needs, improve the efficiency and stability of the server's data processing, and reduce the amount of data called. Then call the selected reference agent for human-computer interaction with the user 130. The process of interaction is that the user 130 inputs the user request information, and the reference agent responds to the user through data interaction with the server. An intelligent teaching system 100 corresponds to multiple servers 120 at the same time, and one server 120 corresponds to one or more terminal devices 110. Each terminal device 110 performs human-computer interaction with a user 130, wherein the number of reference agents corresponds to the number of reference teaching scenarios. The reference teaching scenarios are pre-set according to the different actual application scenarios of users with teacher identities and student identities, and then the corresponding reference agents are set according to the set reference teaching scenarios. The functional type of the preset application function that can be realized by each reference agent is associated with the scenario requirements of the corresponding reference teaching scenarios.

[0017] Based on this, an embodiment of the present application provides a data processing method based on a multimodal intelligent agent, and the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0018] See also Figure 2 , Figure 2 1 is a flow chart of a data processing method based on a multimodal agent provided in an embodiment of the present application. The method is applied to a terminal device 110 in an intelligent teaching system 100, and the intelligent teaching system also includes a server 120. The method includes: Step S201, obtaining the target teaching identity of the user and the target teaching scene in which the user is located.

[0019] Among them, the target teaching identity is used to indicate that the user's corresponding teaching identity is a teacher identity or a student identity, and the target teaching scene is any one of multiple reference teaching scenes that match the teacher identity or the student identity.

[0020] In one possible embodiment, the reference teaching scenarios include a teacher teaching scenario, a teacher lesson preparation scenario, and a student learning scenario; obtaining the user's target teaching identity and the user's target teaching scenario, including: detecting the user's run trigger instruction for the target application, running the target application to display the initial login interface in the target application, and obtaining the agent information stored in the target application; responding to the user's information input operation for the initial login interface to obtain identity authentication information; if the identity authentication information is verified, obtaining the user's corresponding user identity information from a preset local database, and requesting and caching the relevant data of each reference agent according to the interface information; determining the target teaching identity according to the user's corresponding user identity information; and, when it is determined that the target teaching identity is a student identity, determining the target teaching scenario as a student learning scenario; when it is determined that the target teaching identity is a teacher identity, calling the user's teacher course time according to the user identity information, and obtaining the current system time; if the current system time is in the teacher course time, determining the target teaching scenario as the teacher teaching scenario; if the current system time is not in the teacher course time, determining the target teaching scenario as the teacher lesson preparation scenario.

[0021] Among them, the agent information includes interface information for calling the calling interface of the reference agent, the running trigger instruction is generated by a preset trigger event, and the local database is used to store the user identity information associated with all teachers and students in the school where the user is located.

[0022] Among them, in the process of human-computer interaction between the terminal device and the user, the terminal device first detects the user's running trigger instruction for the target application (generated by the preset trigger event), runs the target application and displays the initial login interface. In the process of the terminal device running the target application, the terminal device will obtain the interface information of the reference intelligent agent's calling interface in advance, so as to call each preset reference intelligent agent through the calling interface in advance, improve the speed of the terminal device's subsequent response to the user's identity information to call the target intelligent agent, effectively reduce the response time, and improve teaching efficiency. In addition, in order to ensure the service quality and practicality of intelligent education services, the terminal device will obtain the user's target teaching identity and the corresponding target teaching scene by asking the user to authenticate the identity. It should be noted that this step can also ensure the security of the intelligent teaching system and prevent other off-campus personnel from calling the relevant data of the server in the school's intelligent teaching system.

[0023] Further, for different target teaching identities, the method of determining the target teaching scene is different. The principle is that if the user's target teaching identity is a student, the scene in which the student needs to call the reference agent when he is in or outside the school is usually a student learning scene. If the user's target teaching identity is a teacher, there are different scenes that need to call the reference agent, namely the teacher's lesson preparation scene and the teacher's teaching scene. At this time, the confirmation operation of the target teaching scene can be further performed based on the time when the user starts the target application and the teacher's course time bound to his identity information. The design principle of this embodiment is that based on the detailed analysis of the teaching scene and the user's identity, the user is accurately classified into different teaching scenes by using the identity authentication and time judgment mechanism. Through the interaction between the agent information and the large model base, customized services are provided to users in different scenes, aiming to break the limitations of the traditional teaching system, meet the diversified teaching needs, and improve the overall teaching experience and quality. Compared with the traditional teaching system, the terminal device uses the identity authentication operation as a prerequisite for using the corresponding agent, which can realize accurate push for different scenes and improve the integration and utilization efficiency of teaching resources.

[0024] It can be seen that in this example, the terminal device quickly and accurately identifies the user identity and teaching scene by pre-setting the running trigger instructions and identity authentication, realizes intelligent matching, greatly improves the efficiency of teaching resource allocation, improves the accuracy of scene recognition, and ensures that teaching activities are carried out in an orderly and efficient manner.

[0025] In a possible embodiment, the triggering event includes at least one of the following reference events: detecting a user's touch operation on an optional control in a display interface of a terminal device mapped to a target application; or detecting that the user input voice currently collected contains a running voice instruction for the target application; or detecting that the target feature information of the user action gesture currently collected matches the reference feature information of a preset start gesture for the target application; or detecting that the terminal device is not running any other applications and the current system time is in a high-frequency running period of the target application.

[0026] The reference feature information includes spatial feature information and temporal feature information of a preset start gesture, and the high-frequency operation period is a period in which the operation frequency of the target application program counted within a preset number of days is greater than a preset frequency.

[0027] For example, see Figure 3 , Figure 3 Schematic diagram of a triggering event scenario provided by an embodiment of the present application. Figure 3 As shown, Figure 3 The terminal device in the figure is a mobile phone. The three figures from left to right are respectively the visualization scene diagrams of the interactive scenes corresponding to the three different trigger events supported by the mobile phone. Among them, the first figure represents the user's touch operation on the target application on the display interface of the terminal device. When the terminal device detects the touch operation, it will start the target application to enter the subsequent identity authentication operation; the second figure represents the user staying on the display interface where the target application is displayed, and performing the corresponding control selection operation on the reserved voice input control on the display interface through a finger or a stylus, and the user inputs the running voice command at the same time. Correspondingly, when the terminal device receives the running voice command for the target application, it will parse the command content and prepare to trigger the subsequent operation; the third figure represents the user staying on the display interface where the target application is displayed, and drawing the preset start gesture on the display interface through the gesture input operation. Correspondingly, the terminal device (i.e., the mobile phone) collects the user's action gesture through a camera or sensor, extracts the target feature information of the gesture, and if the target feature information is consistent with the feature information of the preset start gesture, the target application will be started.

[0028] Among them, the trigger event includes a variety of multi-dimensional reference events set based on the commonly used human-computer interaction method between the user and the terminal device. The following introduces the specific monitoring method of the terminal device for the trigger event set in this embodiment: 1. Touch operation monitoring: The system monitors the terminal device display interface in real time and detects the user's touch operation on the optional controls mapped to the target application. Once the relevant touch behavior is captured, the touch position, time and other information are recorded to determine whether it is a valid trigger. 2. Voice command monitoring: With the help of voice recognition technology, the user input voice is continuously collected. The voice content is analyzed to identify whether it contains the running voice command for the target application. If a valid command is recognized, the command content is parsed and the subsequent operation is prepared to be triggered. 3. Gesture feature matching monitoring: The user's action gesture is collected through a camera or sensor, and the target feature information of the gesture is extracted, including spatial features (such as gesture shape, trajectory, angle, etc.) and time features (such as gesture completion time, duration of each stage, etc.). These features are compared with the reference feature information of the preset start gesture for the target application to determine whether they match. 4. Device status and time monitoring: Regularly check the list of applications running on the terminal device to confirm whether other applications are running; at the same time, obtain the current system time and compare it with the high-frequency operation period of the target application obtained by statistics within the preset number of days to determine whether the current time is in this period. Based on the above multi-dimensional monitoring methods, as long as any of the above reference events meets the trigger conditions, the terminal device will immediately generate the corresponding trigger instruction to start the target application.

[0029] Furthermore, the design principle of this embodiment is to use multiple interaction methods such as touch, voice, gesture, etc. through multi-modal interaction fusion to give full play to the advantages of different interaction methods, make up for the limitations of a single interaction method, and realize the diversification and naturalization of human-computer interaction. By providing multiple triggering methods, the operation habits of different users in different scenarios can be met. For example, in a driving scenario, if the user's hands are inconvenient to operate, the navigation application can be quickly started through voice commands; when busy or inconvenient to speak, gesture operation is more convenient, which greatly improves the personalization and comfort of the user experience. In addition, the terminal device can also perform additional data analysis processes to analyze and learn user behavior, automatically start the application according to the high-frequency operation period (that is, statistically analyze the frequency of application use by users within a preset number of days, learn the user's usage habits, and determine the high-frequency operation period) and the device operation status (that is, real-time perception of contextual information such as device operation status and current time, combined with the user behavior analysis results, to make intelligent decisions), to achieve intelligent and automated operation. The system can learn the user's usage habits and automatically start the application when the user is most likely to use it, reducing the user's manual operation steps and improving the efficiency of use. In addition, since multi-dimensional trigger events are set up, the terminal device can quickly capture the user's intention to launch the application through accurate recognition of multi-dimensional information such as touch, voice, gestures, and accurate judgment of device status and time, so that the application can respond quickly, shorten user waiting time, and improve the system's response speed and user service experience.

[0030] It can be seen that in this example, the terminal device provides users with a variety of triggering methods for triggering target applications through multi-modal interaction fusion, so as to meet the habits of different users, thereby improving user operation convenience and reducing time loss. In addition, the terminal device can automatically trigger according to the device operating status and high-frequency operating time period, improving the intelligence of the teaching system and the continuity of teaching services. In addition, by combining the trigger event with subsequent identity and scene recognition, users can be quickly matched to the appropriate teaching scene, further improving the efficiency of teaching resource allocation and the accuracy of scene recognition.

[0031] Step S202, determining and calling a reference agent corresponding to the target teaching scenario as the target agent.

[0032] Among them, each reference intelligent agent corresponds to a reference teaching scene. The reference intelligent agent is used to interact with the user to determine multiple preset application functions required to be used in the corresponding reference teaching scene. The reference intelligent agent is communicated with the large model base pre-set in the server.

[0033] Step S203, collecting user request information input by the user through the target agent, and based on the data interaction operation between the target agent and the large model base, to determine the target response result corresponding to the target application function.

[0034] The target application function is determined by user request information, and the target application function is any one of a plurality of preset application functions.

[0035] In a possible embodiment, user request information input by the user is collected through the target agent, and a target response result corresponding to the target application function is determined based on the data interaction operation between the target agent and the large model base, including: collecting content data input by the user through the target agent, and determining the target input format corresponding to the content data; determining the user request information based on the content data and the target input format; sending a result request message carrying the user request information to the corresponding large model base through the target agent; and receiving the target response result sent by the output control module.

[0036] Among them, the result request message is used to instruct the big model base to call the built-in module to perform data analysis operations on the user request information to return the target response result and target application function. The built-in module includes a dedicated big model, a local knowledge base, a third-party plug-in and an output control module. The local knowledge base is used to store the teaching materials corresponding to various subjects in the user's school. The dedicated big model is used to combine the local knowledge base to parse and content data according to the target input format. The third-party plug-in is used to call the teaching tools associated with multiple preset application functions corresponding to the target teaching scenario. The input control module is used to integrate the sub-response results output by the dedicated big model, local knowledge base and third-party plug-in into the target response result according to the target application function indicated by the user request information.

[0037] For example, see Figure 4 , Figure 4 Schematic diagram of the architecture of a target intelligent agent and a large model base provided in an embodiment of the present application. Figure 4As shown, the user performs a user input operation on the terminal device to input the user input information corresponding to the target input format, wherein the target input format is any one of the multiple reference input formats supported by the large model base, and the types of reference input formats include but are not limited to voice, text, image and video. Among them, the voice part uses a digital microphone to collect audio, the text uses an input method to collect user text input, and the image and video use a camera input. Because the multimodal large model accepts user input in multiple formats, the target agent can directly send the user input information that conforms to the reference input format directly to the large model for processing without additional format conversion and processing. Further, the terminal device calls the target agent (divided into teacher preparation agent, teacher teaching agent and student learning agent according to the target teaching scenario) as a human-computer interaction interface, is responsible for collecting the content data input by the user, determining its corresponding target input format, and sending a result request message carrying the user request information to the large model base. Among them, the large model base includes: a dedicated large model: parses the content data according to the target input format, and analyzes and processes it in combination with the data provided by the local knowledge base. Local knowledge base: stores the teaching materials corresponding to various subjects of the school, and provides data support for the dedicated large model and output control module. Third-party plug-in: Calls the teaching tools associated with the preset application functions of the target teaching scenario to assist in generating response results. Output control module: According to the target application function indicated by the user request information, integrates the sub-response results output by each module, forms the target response result and returns it to the target intelligent agent. Through the above-mentioned system architecture of the target intelligent agent and the large model base, the terminal device can interact with the large model base in the server by calling the target intelligent agent to realize the intelligent teaching service required by the user.

[0038] Among them, in this embodiment, the interaction between the target agent called by the terminal device and the server can realize multi-source data integration and processing, as well as function-oriented result generation. Among them, multi-source data integration and processing refers to the server collecting the content data input by the user through obtaining the target agent, and adapting it according to the reference input format supported by the large model base to ensure that the data can be effectively processed. Then, by integrating the capabilities of the dedicated large model, local knowledge base and third-party plug-ins, using the dedicated large model to parse the data, combining the local knowledge base to provide teaching material support, calling third-party plug-ins to obtain relevant teaching tools, multi-source data fusion processing is realized to meet the diverse needs of users. Among them, function-oriented result generation refers to determining the target application function based on user request information. Focusing on the target application function, the output control module integrates the sub-response results output by each part to generate the final target response result, thereby achieving precise matching from user needs to function realization.

[0039] Specifically, based on Figure 4The system architecture of the target agent and the large model base shown is aimed at the teacher teaching scenario, the teacher preparation scenario and the student learning scenario. The specific user use scenario of the intelligent teaching service through the terminal device is as follows: 1. Teacher teaching scenario: When teaching, the teacher inputs content data such as knowledge point explanation requirements and interactive link design through the target agent. The system determines the target application function based on the input, such as generating teaching courseware, designing classroom questions, etc. The dedicated large model combines the teaching materials in the local knowledge base, and third-party plug-ins such as interactive game tools are called, and the output control module integrates the results to provide teachers with complete teaching auxiliary content. 2. Teacher preparation scenario: The teacher inputs the course theme, teaching objectives and other content, and the system determines the target application function as the collection and collation of preparation materials. The large model parses the content, obtains relevant information from the local knowledge base, and the third-party plug-in calls the lesson plan template, material search tool, etc., and finally outputs the control module to integrate and generate a detailed preparation plan. 3. Student learning scenario: Students input content data such as learning questions and homework problems, and the system determines the target application function as answering questions and solving doubts. The dedicated large model combines answers from the knowledge base, third-party plug-ins call problem-solving analysis tools, and the output control module integrates and generates detailed answer content and learning suggestions.

[0040] It can be seen that in this example, the terminal device calls the target agent to accurately collect user data, and then matches the large model input format so that the large model base in the server can quickly understand the user's needs. In addition, through the collaboration of the built-in modules of the large model base, data is parsed, tools are called, and results are integrated to meet the needs of various teaching functions and improve the practicality of the terminal device and the efficiency of data processing.

[0041] Step S204, after the target agent receives the target response result returned by the large model base, it implements the target application function according to the target response result.

[0042] In one possible embodiment, the target teaching scenario is a student learning scenario, and the target application function is an after-class exercise function; after the target intelligent agent receives the target response result returned by the large model base, the target application function is implemented according to the target response result, including: determining the user's current grade and the course progress corresponding to the target subject according to the user identity information; performing a content screening operation on the target response result according to the current grade and the course progress to determine reference screening results of multiple difficulty levels; according to the preset exercise strategy, outputting the reference screening results corresponding to the first level, the second level and the third level respectively, so as to display the corresponding after-class exercises in sequence; and detecting the user's answer input operation for the after-class exercises to obtain the user's answer to each after-class exercise; determining the reference answer to each after-class exercise according to the target response result and the multiple reference screening results; outputting the exercise correction result according to the user's answer to the after-class exercise and the reference answer to the after-class exercise.

[0043] Among them, the reference screening results are used to display the homework questions that match the corresponding difficulty levels. The difficulty levels of the homework questions represented by each difficulty level are different. The difficulty levels include the first level, the second level and the third level. The difficulty levels of the homework questions represented by the difficulty levels are the first level, the second level and the third level from high to low. The exercise correction results are used to display the user's current score information for the homework questions and the answer analysis content of the homework questions with incorrect answers.

[0044] Among them, in this embodiment, the target response result can be understood as a comprehensive result generated by the large model base after a series of processing for the user request information, which is the key data basis for realizing the after-class exercise function. The large model base generates the target response result associated with the after-class exercise function through the collaborative work of multiple modules. Specifically, the target response result covers various types of information related to the after-class exercise function, which can meet the needs of each link of subsequent exercises. For example, it may contain exercise resources of different difficulty levels, which may be selected from the local knowledge base according to conditions such as student grade and course progress, and analyzed and sorted by a dedicated large model; it may also contain ideas for solving exercises, knowledge point prompts and other content, so as to generate reference answers and answer analysis. It provides a basis for the subsequent steps of realizing the after-class exercise function. According to the student's grade and course progress, the target response result is content screened, and the reference screening results of different difficulty levels are determined, and then the corresponding after-class exercises are output. At the same time, the target response result also plays an important guiding role in determining the reference answers to the exercises and generating the results of exercise correction.

[0045] Among them, in this embodiment, after receiving the target response result returned by the large model base, the target intelligent body will first determine the identity and scene of the user, that is, based on the user identity information, accurately locate the student's current grade and the course progress of the target subject. Further, the target intelligent body will combine the content in the target response result with the determined grade and course progress, and perform content screening operations on the target response result, filter out content that fits different levels of difficulty, and generate reference screening results of the first level, second level and third level, which correspond to high, medium and low difficulty after-school exercises respectively. Subsequently, according to the preset practice strategy, different levels of exercises are displayed to students in turn. Finally, the target intelligent body realizes the student answering process through interaction with the user, and after the student completes the answer, the target intelligent body replaces the identity of the teacher to correct the after-school exercises, and practices the correction results, which include score information and detailed answer analysis content of exercises with wrong answers.

[0046] Furthermore, compared with traditional educational services, the target agent in this embodiment can provide matching learning resources and practice content based on individual differences of students, so that each student can fully develop in his or her own zone of proximal development, and improve the pertinence and effectiveness of learning. In addition, it can replace the work of teachers, realize automatic correction of homework, quickly generate scores and analysis of wrong questions, so that students can understand their learning status in time and adjust their learning strategies; teachers can devote their energy to teaching guidance and improve teaching efficiency. In addition, by setting exercises of different difficulty levels, students are guided to gradually improve their knowledge and skills, avoiding questions that are too difficult or too easy, and improving user experience.

[0047] It can be seen that in this example, the terminal device uses the target agent to determine the grade and course progress based on the user's identity information, accurately screen the after-class exercise content, generate exercises of multiple difficulty levels to meet the learning needs of different students, and improve the flexibility of the agent to achieve the target application function. In addition, the answers are detected in real time and the correction results are given to help students understand their learning situation; and the target agent can automatically complete the screening, correction and analysis, reducing the burden on teachers and improving the level of intelligent teaching and teaching quality.

[0048] In one possible embodiment, the target teaching scene is a student learning scene, and the target intelligent agent also includes an emotion recognition module; after determining the exercise correction result according to the user's answer to the after-class exercise and the reference answer to the after-class exercise, the method also includes: collecting the user's audio and video information in the answer input operation through the emotion recognition module; determining a first emotion recognition result according to the intonation feature and the speech speed feature in the voice input information, and determining a second emotion recognition result according to the facial expression feature vector and the body movement timing feature in the image input information; fusing the first emotion recognition result and the second emotion recognition result according to a preset fusion algorithm to generate a current emotion state evaluation result; points; and, if the negative dimension score corresponding to the negative dimension exceeds the first score threshold, the preset motivational corpus in the local knowledge base is called to generate and display personalized encouragement information, and the difficulty level of the after-class exercises displayed next time is reduced; or, if the negative dimension score is detected to exceed the second score threshold for consecutive preset warning times, the device information of the user's guardian's terminal device is retrieved through a third-party plug-in, and the second score threshold is greater than the first score threshold; the emotional fluctuation curve is determined according to the current emotional state score corresponding to the warning number, and the learning status information is determined according to the exercise correction results corresponding to the warning number; and a warning report is sent to the guardian's terminal device according to the device information.

[0049] Among them, audio and video information includes voice input information and image input information. The current emotional state score is used to characterize the degree of emotional tendency in the positive dimension, neutral dimension and negative dimension reflected by the user in the current state. The early warning report carries the emotional fluctuation curve and learning status information.

[0050] Among them, the target teaching scenario applicable to this embodiment is the student learning scenario, the target application function is the after-class exercise function, and the target intelligent body first determines the exercise correction result based on the user's answer and the reference answer to the after-class exercise. After that, the target intelligent body will first collect emotional information, that is, collect audio and video information during the student's learning process while the student is inputting the answer. After that, the intonation features and speech speed features are extracted from the voice input information in the audio and video information, and the first emotion recognition result is determined based on this; the facial expression feature vector and the body movement timing features are extracted from the image input information to determine the second emotion recognition result. Then, these two results are fused according to the preset fusion algorithm to generate a current emotional state score for characterizing the degree of emotional tendency of the student in the positive, neutral and negative dimensions. Finally, the target intelligent body analyzes the current emotional state score to ensure the enthusiasm of the student user to learn. In the last step, if the target agent detects that the negative dimension score corresponding to the negative dimension exceeds the first score threshold (that is, the student is in a negative mood at this time), the preset motivational corpus in the local knowledge base is called to generate and display personalized encouragement information to the student, and at the same time reduce the difficulty level of the next after-class exercises displayed, thereby improving the student's learning enthusiasm and the confidence of subsequent students in learning.

[0051] Furthermore, the terminal device is also provided with a guardian warning mechanism, that is, if the negative dimension score exceeds the second score threshold (the second score threshold is greater than the first score threshold) for consecutive preset warning times, the device information of the student guardian terminal device is retrieved through a third-party plug-in. Thus, the emotional fluctuation curve of the student user during learning and the learning status information determined by the exercise correction results are sent to the guardian together, so that the guardian can understand the student's situation in time and conduct manual intervention. Compared with traditional teaching, the data processing process of the target intelligent body in this embodiment not only focuses on the evaluation of the learning outcomes of the student's learning itself, but also adds real-time monitoring of the emotional state during the learning process, and improves the attention to the physical and mental health of the students. In addition, by notifying the guardian in time when the student's emotions continue to be negative, providing detailed emotional fluctuation curves and learning status information, it is convenient for the guardian to understand the student's situation, collaborate with the school, take more effective educational measures, form a joint force of home-school education, and improve the flexibility of the intelligent teaching system in performing data processing.

[0052] In one possible embodiment, the intonation features and speech rate features of the speech signal are extracted, and a pre-trained speech emotion classification model is used to determine the first emotion recognition result; the facial expression feature vector and the body movement timing features are extracted, and a second emotion recognition result is determined through a convolutional neural network and a timing classification model.

[0053] Among them, the specific implementation method of determining the first emotion recognition result is as follows: the terminal device first pre-processes the collected voice signal to remove interference such as background noise to improve the signal quality. Subsequently, digital signal processing technology is used to extract intonation features from the pre-processed voice signal, such as the fluctuation of pitch, the rise and fall pattern of tone, etc.; at the same time, the speed features are determined by calculating the number of syllables or words in the voice per unit time. The extracted intonation and speed features are used as input and sent to the pre-trained voice emotion classification model. The model is based on a deep learning algorithm and is trained with a large amount of voice data with emotion annotations, so as to output the first emotion recognition result and judge the emotional tendency reflected in the student's voice.

[0054] Among them, the specific implementation method of determining the second emotion recognition result is as follows: for the collected image information, using computer vision technology, with the help of a special facial feature extraction algorithm, extract the facial expression feature vector, accurately capture the details of the expression such as eyebrows rising and mouth corners drooping. At the same time, the motion capture technology is used to analyze the sequence of body movements, speed changes, etc., to determine the temporal characteristics of body movements. The facial expression feature vector and the temporal characteristics of body movements are input into the model architecture composed of a convolutional neural network and a temporal classification model. The convolutional neural network is good at processing image spatial features and can effectively extract the key features of facial expressions; the temporal classification model can process body movement data with temporal characteristics and mine the temporal dependencies therein. The two work together to finally output the second emotion recognition result and judge the student's emotional state from a visual perspective.

[0055] It can be seen that in this example, the emotion recognition module is used to collect students' audio and video information, analyze multi-dimensional features to determine the emotional state, and dynamically adjust the learning strategy according to the degree of negative emotions. When the situation is mild, encouragement is given and the difficulty is reduced. When the situation is serious, the guardian's contact information can be called to conduct home-school linkage. If the situation continues to be negative, a warning report including emotions and learning status will be sent to the guardian, thereby comprehensively optimizing the learning experience and improving the practicality of the specific application functions realized by the intelligent body.

[0056] In a possible embodiment, the target teaching scene is a teacher teaching scene, the target application function is a classroom auxiliary function, the target intelligent body also includes an emotion recognition module, and the intelligent teaching system also includes a classroom terminal in the classroom corresponding to the teacher teaching scene; after the target intelligent body receives the target response result returned by the large model base, the target application function is realized according to the target response result, including: determining the classroom recording video collected in real time for the teacher teaching scene according to the target response result; performing video analysis operations according to the classroom recording video to determine the facial expression data corresponding to the students and the body movement data corresponding to the teacher; determining the class attention score according to the facial expression data, and The body movement data is used to determine the teacher's teaching stress value; and, if the class attention score is lower than the third score threshold for a duration greater than a preset duration, a third-party plug-in is called through the target intelligent agent to push in-class test questions or group discussion tasks to the classroom terminal; or, if the teacher's teaching stress value is greater than or equal to the fourth score threshold, the local knowledge base is called through the target intelligent agent to match the teaching design case of the target subject, and obtain the classroom lesson plan of the current teacher teaching scenario uploaded in advance by the user; according to the teaching design case, the current classroom lesson plan is updated so that the teaching difficulty level of the updated classroom lesson plan is lower than the teaching difficulty level of the classroom lesson plan before the update.

[0057] Among them, the class attention score is used to represent the average value of attention concentration in the target class recorded in the classroom recording video, the teacher's teaching stress value is used to represent the user's tension in the current teacher teaching scenario, and the in-class test questions and group discussion tasks are associated with the target subject corresponding to the current teacher teaching scenario.

[0058] Among them, the target response result is the content returned by the large model base after a series of data analysis and processing based on the request information sent by the target agent. In this embodiment, it is an important basis for realizing classroom auxiliary functions, and contains key information related to the teacher's teaching scene, such as the identifier or data path pointing to the real-time classroom recording video. This information enables the target agent to determine and obtain the classroom recording video, and then carry out a series of operations such as subsequent video analysis, indicator calculation, and implementation of targeted measures to achieve various requirements of classroom auxiliary functions.

[0059] For example, see Figure 5 , Figure 5 Schematic diagram of a teacher teaching scenario provided in an embodiment of the present application. Figure 5As shown, the camera in the classroom where the teacher teaches will take real-time classroom recording videos corresponding to each class and upload them to the large model base in the server in the intelligent teaching system. After the user whose teaching identity is a teacher calls the teacher teaching intelligent body and enables the target application function of the classroom auxiliary function, the teacher teaching intelligent body will call the classroom recording video in the large model base and perform corresponding video analysis operations on students and teachers respectively. Specifically, through the video analysis operation, facial expressions of students are collected to obtain facial expression data, and body movement data of teachers are collected to obtain body movement data, and then the subsequent steps in this embodiment are executed, that is, the class attention distribution is determined by facial expression data and the teacher's teaching pressure value is determined according to the body movement data, and classroom auxiliary operations are performed on the teacher's teaching according to the classroom evaluation of these two dimensions.

[0060] Among them, the design principle of this embodiment is to quantify students' attention (i.e., conduct an evaluation operation of class attention score) and teachers' teaching pressure (i.e., conduct an evaluation operation of teachers' teaching pressure value) by analyzing the facial expressions and body movement data in the classroom recording video, and make decisions based on these data to make teaching adjustments more scientific and targeted; and, through the collaborative work of multiple modules such as target intelligent agents, third-party plug-ins and local knowledge bases, integrate different resources and functions to achieve comprehensive support and optimization of complex teaching scenarios (i.e., for different teaching situations, such as students' lack of concentration or teachers' excessive pressure, provide personalized solutions, such as pushing appropriate tasks or adjusting the difficulty of teaching plans, to meet the diverse needs in actual teaching); by paying attention to the status of teachers and students in the classroom, providing support to teachers through technical means, helping teachers better cope with teaching challenges, and improving students' learning experience.

[0061] It can be seen that in this example, with the help of the emotion recognition module, the facial expressions of students and the body movements of teachers in the classroom recording video are analyzed to accurately grasp the class attention score and the teacher's teaching pressure value, and the teaching strategy is dynamically optimized based on this to attract students' attention, reduce teacher pressure, and improve the quality of classroom teaching.

[0062] It can be seen that by building in different types of reference agents and setting up a large model base in the server, the server can provide corresponding intelligent assistance for different links in the teaching process, thereby improving the intelligence and practicality of the teaching system. In addition, the terminal device can access the large model through the agent to improve the efficiency and flexibility of teachers' teaching and students' learning, as well as improve the flexibility of the agent in processing the data to be processed by the user and improve the user's feedback experience.

[0063] The following is an embodiment of the device of the present application. The embodiment of the device of the present application and the embodiment of the method of the present application are of the same concept and are used to execute the method described in the embodiment of the present application. For the convenience of explanation, the embodiment of the device of the present application only shows the part related to the embodiment of the device of the present application. For the specific technical details not disclosed, please refer to the description of the embodiment of the method of the present application, which will not be repeated here one by one.

[0064] The embodiment of the present application provides a data processing device based on a multimodal agent, and the data processing device based on a multimodal agent is applied to Figure 1 The terminal device 110 in the intelligent teaching system 100 shown in the figure, the intelligent teaching system 100 also includes a server 120. Specifically, the multimodal agent-based data processing device is used to execute the steps executed by the terminal device in the above multimodal agent-based data processing method. The multimodal agent-based data processing device provided in the embodiment of the present application may include modules corresponding to the corresponding steps.

[0065] The embodiment of the present application can divide the functional modules of the data processing device based on the multimodal agent according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. The division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0066] In the case of dividing each functional module into corresponding functional modules, Figure 61 is a functional unit composition block diagram of a data processing device based on a multimodal agent provided in an embodiment of the present application; the data processing device based on a multimodal agent 60 is applied to a terminal device 110 in an intelligent teaching system 100, and the intelligent teaching system 100 also includes a server 120; the data processing device based on a multimodal agent 60 includes: an information acquisition unit 601, which is used to acquire the target teaching identity of the user and the target teaching scene in which the user is located, the target teaching identity is used to indicate that the teaching identity corresponding to the user is a teacher identity or a student identity, and the target teaching scene is any one of a plurality of reference teaching scenes matching the teacher identity or the student identity; an agent calling unit 602, which is used to determine and call a reference agent corresponding to the target teaching scene as the target agent The reference agent is used to interact with the user to determine a plurality of preset application functions required to be used in the corresponding reference teaching scenario. The reference agent is connected to the large model base pre-set in the server in communication; a data interaction unit 603 is used to collect user request information input by the user through the target agent, and to determine the target response result corresponding to the target application function based on the data interaction operation between the target agent and the large model base. The target application function is determined by the user request information, and the target application function is any one of a plurality of preset application functions; a function realization unit 604 is used to realize the target application function according to the target response result after the target agent receives the target response result returned by the large model base.

[0067] In one possible embodiment, the reference teaching scenarios include teacher teaching scenarios, teacher lesson preparation scenarios, and student learning scenarios; in terms of obtaining the user's target teaching identity and the user's target teaching scenario, the information acquisition unit 601 is specifically used to: detect the user's run trigger instruction for the target application, run the target application to display the initial login interface in the target application, and obtain the agent information stored in the target application, the agent information includes interface information for calling the calling interface of the reference agent, and the run trigger instruction is generated by a preset trigger event; respond to the user's information input operation on the initial login interface to obtain the identity authentication information. If the identity authentication information is verified, it is obtained from the preset local database. Obtain the user identity information corresponding to the user, and according to the interface information, request and cache the relevant data of each reference agent, the local database is used to store the user identity information associated with all teachers and students in the school where the user is located; determine the target teaching identity according to the user identity information corresponding to the user; and, when the target teaching identity is determined to be a student identity, determine the target teaching scene to be a student learning scene; when the target teaching identity is determined to be a teacher identity, call the user's teacher course time according to the user identity information, and obtain the current system time; if the current system time is in the teacher course time, determine the target teaching scene to be the teacher teaching scene; if the current system time is not in the teacher course time, determine the target teaching scene to be the teacher lesson preparation scene.

[0068] In a possible embodiment, the trigger event includes at least one of the following reference events: detecting a user's touch operation on an optional control in a display interface of a terminal device mapped to a target application; or detecting that the user input voice currently collected contains a voice instruction for running the target application; or detecting that the target feature information of the user action gesture currently collected matches the reference feature information of a preset start gesture for the target application, and the reference feature information includes spatial feature information and temporal feature information of the preset start gesture; or detecting that the terminal device is not running other applications and the current system time is in a high-frequency operation period of the target application, and the high-frequency operation period is a period of time when the operation frequency of the target application counted within a preset number of days is greater than the preset frequency.

[0069] In one possible embodiment, in terms of collecting user request information input by the user through the target agent, and determining the target response result corresponding to the target application function based on the data interaction operation between the target agent and the large model base, the data interaction unit 603 is specifically used to: collect content data input by the user through the target agent, and determine the target input format corresponding to the content data, where the target input format is any one of multiple reference input formats supported by the large model base; determine the user request information based on the content data and the target input format; send a result request message carrying the user request information to the corresponding large model base through the target agent, wherein the result request message is used to instruct the large model base to call the built-in module Data analysis operations are performed on user request information to return target response results and target application functions. Built-in modules include a dedicated large model, a local knowledge base, a third-party plug-in, and an output control module. The local knowledge base is used to store teaching materials corresponding to various subjects in the user's school. The dedicated large model is used to combine the local knowledge base to parse and process content data according to the target input format. The third-party plug-in is used to call the teaching tools associated with multiple preset application functions corresponding to the target teaching scenario. The input control module is used to integrate the sub-response results output by the dedicated large model, the local knowledge base, and the third-party plug-in into the target response result according to the target application function indicated by the user request information; and receive the target response result sent by the output control module.

[0070] In one possible embodiment, the target teaching scenario is a student learning scenario, and the target application function is an after-class exercise function; after the target intelligent agent receives the target response result returned by the large model base, in terms of realizing the target application function according to the target response result, the data interaction unit 603 is specifically used to: determine the user's current grade and the course progress corresponding to the target subject according to the user identity information; perform a content screening operation on the target response result according to the current grade and course progress to determine reference screening results of multiple difficulty levels, and the reference screening results are used to display after-class exercises that match the corresponding difficulty levels. The difficulty of the after-class exercises represented by each difficulty level is different, and the difficulty levels include the first level, the second level and the third level. The difficulty of the homework questions represented by the difficulty level is from high to low as the first level, the second level and the third level; according to the preset exercise strategy, the reference screening results corresponding to the first level, the second level and the third level are output respectively to display the corresponding homework questions in sequence; and, the user's answer input operation for the homework questions is detected to obtain the user's answer to each homework question; according to the target response result and multiple reference screening results, the reference answer to each homework question is determined; according to the user answer to the homework question and the reference answer to the homework question, the exercise correction result is output, and the exercise correction result is used to display the user's current homework score information and the answer analysis content of the homework questions with wrong answers.

[0071] In one possible embodiment, the target teaching scenario is a student learning scenario, and the target intelligent agent also includes an emotion recognition module; after determining the exercise correction result based on the user's answer to the after-class exercises and the reference answer to the after-class exercises, the function implementation unit 604 is specifically used to: collect the user's audio and video information in the answer input operation through the emotion recognition module, the audio and video information includes voice input information and image input information; determine the first emotion recognition result based on the intonation characteristics and speech speed characteristics in the voice input information, and determine the second emotion recognition result based on the facial expression feature vector and body movement timing characteristics in the image input information; fuse the first emotion recognition result and the second emotion recognition result according to a preset fusion algorithm to generate a current emotion state score, and the current emotion state score is used to characterize the user's current emotion state score. The emotional tendency degree of the positive dimension, neutral dimension and negative dimension reflected in the state; and, if the negative dimension score corresponding to the negative dimension exceeds the first score threshold, the preset motivational corpus in the local knowledge base is called to generate and display personalized encouragement information, and the difficulty level of the after-class exercises displayed next time is reduced; or, if the negative dimension score is detected to exceed the second score threshold for the consecutive preset warning times, the device information of the user's guardian's terminal device is retrieved through a third-party plug-in, and the second score threshold is greater than the first score threshold; the emotional fluctuation curve is determined according to the current emotional state score corresponding to the warning times, and the learning status information is determined according to the exercise correction results corresponding to the warning times; a warning report is sent to the guardian's terminal device according to the device information, and the warning report carries the emotional fluctuation curve and learning status information.

[0072] In one possible embodiment, the target teaching scene is a teacher teaching scene, the target application function is a classroom auxiliary function, the target intelligent body also includes an emotion recognition module, and the intelligent teaching system also includes a classroom terminal in the classroom corresponding to the teacher teaching scene; after the target intelligent body receives the target response result returned by the large model base, the target application function is realized according to the target response result, and the function realization unit 604 is specifically used to: determine the classroom recording video collected in real time for the teacher teaching scene according to the target response result; perform video analysis operations according to the classroom recording video to determine the facial expression data corresponding to the students and the body movement data corresponding to the teacher; determine the class attention score according to the facial expression data, and determine the teacher's teaching pressure value according to the body movement data, and the class attention score is used to characterize the attention in the target class recorded by the classroom recording video. The average value of concentration, the teacher's teaching stress value is used to characterize the user's tension in the current teacher teaching scenario; and, if the class attention score is lower than the third score threshold for a duration greater than the preset duration, the target intelligent agent calls a third-party plug-in to push in-class test questions or group discussion tasks to the classroom terminal, and the in-class test questions and group discussion tasks are associated with the target subject corresponding to the current teacher teaching scenario; or, if the teacher's teaching stress value is greater than or equal to the fourth score threshold, the target intelligent agent calls the local knowledge base to match the teaching design case of the target subject, and obtains the classroom lesson plan of the current teacher teaching scenario uploaded in advance by the user; according to the teaching design case, the current classroom lesson plan is updated so that the teaching difficulty level of the updated classroom lesson plan is lower than the teaching difficulty level of the classroom lesson plan before the update.

[0073] In the case of integrated units, such as Figure 7 As shown, Figure 7 1 is a block diagram of the functional units of another data processing device based on a multimodal agent provided in an embodiment of the present application. Figure 7 In the embodiment, the data processing device 60 based on the multimodal agent includes: a processing module 720 and a communication module 710. The processing module 720 is used to control and manage the actions of the data processing device 60 based on the multimodal agent, for example, the steps of the information acquisition unit 601, the agent calling unit 602, the data interaction unit 603 and the function realization unit 604, and / or other processes for executing the technology described herein. The communication module 710 is used to support the interaction between the data processing device based on the multimodal agent and other devices. Figure 7 As shown, the data processing device based on the multimodal agent may include a storage module 730, and the storage module 730 is used to store the program code and data of the data processing device based on the multimodal agent.

[0074] Among them, the processing module 720 can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 710 can be a transceiver, an RF circuit or a communication interface, and the like. The storage module 730 can be a memory.

[0075] Among them, all relevant contents of each scenario involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. The above multimodal agent-based data processing device 60 can execute the above Figure 2 The multimodal agent-based data processing method shown.

[0076] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center containing one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0077] Figure 8 is a structural block diagram of a terminal device provided in an embodiment of the present application. Figure 8As shown, the terminal device 110 may include one or more of the following components: a processor 810, a memory 820 coupled to the processor 810, wherein the memory 820 may store one or more computer programs 821, and the one or more computer programs 821 may be configured to implement the methods described in the above embodiments when executed by one or more processors 810. The server here is the terminal device 110 in the above embodiments.

[0078] The processor 810 may include one or more processing cores. The processor 810 uses various interfaces and lines to connect the various parts of the entire terminal device 110, and executes various functions and processes data of the terminal device 110 by running or executing instructions, programs, code sets or instruction sets stored in the memory 820, and calling data stored in the memory 820. Optionally, the processor 810 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 810 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 810, but may be implemented separately through a communication chip.

[0079] The memory 820 may include a random access memory (RAM) or a read-only memory (ROM). The memory 820 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the terminal device 110 during use, etc.

[0080] It is understandable that the terminal device 110 may include more or fewer structural elements than those in the above structural block diagram, which is not limited here.

[0081] An embodiment of the present application also provides a computer storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, part or all of the steps of any method recorded in the above method embodiments are implemented.

[0082] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0083] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0084] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are only schematic; for example, the division of units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0085] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0086] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0087] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a volatile memory or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM) and direct RAM bus random access memory (DR RAM), among other media that can store program code.

[0088] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.

Claims

1. A data processing method based on multimodal agents, characterized in that: A terminal device applied to an intelligent teaching system, wherein the intelligent teaching system further comprises a server; the method comprises: Acquire a target teaching identity of a user and a target teaching scenario of the user, wherein the target teaching identity is used to indicate that the teaching identity corresponding to the user is a teacher identity or a student identity, and the target teaching scenario is any one of a plurality of reference teaching scenarios matching the teacher identity or the student identity; Determine and call a reference agent corresponding to the target teaching scene as the target agent, each of the reference agents corresponds to one reference teaching scene, the reference agent is used to interact with the user to determine a plurality of preset application functions required to be used in the corresponding reference teaching scene, and the reference agent is communicatively connected with a large model base pre-set in the server; The target agent collects the user request information input by the user, and determines the target response result corresponding to the target application function based on the data interaction operation between the target agent and the large model base, wherein the target application function is determined by the user request information and the target application function is any one of the multiple preset application functions; After the target agent receives the target response result returned by the large model base, the target application function is implemented according to the target response result.

2. The method according to claim 1, characterized in that The reference teaching scenarios include teacher teaching scenarios, teacher lesson preparation scenarios, and student learning scenarios; the step of obtaining the target teaching identity of the user and the target teaching scenario in which the user is located includes: Detecting a run trigger instruction from the user for a target application, running the target application to display an initial login interface in the target application, and acquiring agent information stored in the target application, the agent information including interface information for calling a calling interface of the reference agent, wherein the run trigger instruction is generated by a preset trigger event; In response to the user's information input operation on the initial login interface, obtaining identity authentication information; If the identity verification information is verified, the user identity information corresponding to the user is obtained from a preset local database, and according to the interface information, the relevant data of each reference agent is requested and cached, and the local database is used to store the user identity information associated with all teachers and students in the school where the user is located; Determining the target teaching identity according to the user identity information corresponding to the user; and, When it is determined that the target teaching identity is the student identity, the target teaching scene is determined to be the student learning scene; When it is determined that the target teaching identity is the teacher identity, calling the teacher course time of the user according to the user identity information, and obtaining the current system time; If the current system time is in the teacher's class time, determining the target teaching scene to be the teacher's teaching scene; If the current system time is not in the teacher's class time, the target teaching scene is determined to be the teacher's lesson preparation scene.

3. The method according to claim 2, characterized in that The trigger event includes at least one of the following reference events: Detecting a touch operation of the user on an optional control mapped by the target application to a display interface of the terminal device; or, Detecting that the currently collected user input voice contains a running voice instruction for the target application; or, Detecting that the target feature information of the user action gesture currently collected matches the reference feature information of the preset start gesture for the target application, the reference feature information including the spatial feature information and the temporal feature information of the preset start gesture; or, It is detected that the terminal device is not running other applications, and the current system time is in a high-frequency operation period of the target application, and the high-frequency operation period is a period in which the operation frequency of the target application counted within a preset number of days is greater than a preset frequency.

4. The method according to claim 3, characterized in that: The collecting of the user request information input by the user through the target agent, and determining the target response result corresponding to the target application function based on the data interaction operation between the target agent and the large model base, include: Collecting the content data input by the user through the target agent, and determining a target input format corresponding to the content data, wherein the target input format is any one of a plurality of reference input formats supported by the large model base; Determining the user request information according to the content data and the target input format; A result request message carrying the user request information is sent to the corresponding large model base through the target agent, wherein the result request message is used to instruct the large model base to call a built-in module to perform a data analysis operation on the user request information to return the target response result and the target application function, and the built-in module includes a dedicated large model, a local knowledge base, a third-party plug-in and an output control module, the local knowledge base is used to store the teaching materials corresponding to various subjects in the school where the user is located, the dedicated large model is used to combine the local knowledge base to parse the content data according to the target input format, the third-party plug-in is used to call the teaching tools associated with the multiple preset application functions corresponding to the target teaching scenario, and the input control module is used to integrate the sub-response results output by the dedicated large model, the local knowledge base and the third-party plug-in into the target response result according to the target application function indicated by the user request information; Receive the target response result sent by the output control module.

5. The method according to claim 4, characterized in that The target teaching scenario is the student learning scenario, and the target application function is an after-class exercise function; after the target agent receives the target response result returned by the large model base, the target application function is realized according to the target response result, including: Determine the course progress corresponding to the current grade and target subject of the user according to the user identity information; According to the current grade and the course progress, a content screening operation is performed on the target response result to determine reference screening results of multiple difficulty levels, the reference screening results are used to display after-class exercises matching the corresponding difficulty levels, the difficulty levels of the after-class exercises represented by each difficulty level are different from each other, the difficulty levels include a first level, a second level and a third level, and the difficulty levels of the after-class exercises represented by the difficulty levels are, from high to low, the first level, the second level and the third level; According to a preset practice strategy, the reference screening results corresponding to the first level, the second level and the third level are output respectively, so as to display the corresponding after-class exercises in sequence; and, Detecting the user's input operation of answering the after-class exercises to obtain the user's answer to each of the after-class exercises; Determining a reference answer for each of the after-class exercises according to the target response result and the multiple reference screening results; Output the exercise marking result according to the user's answer to the exercise and the reference answer to the exercise, wherein the exercise marking result is used to display the score information of the user's current exercise and the answer analysis content of the exercise with wrong answer.

6. The method according to claim 5, characterized in that The target teaching scenario is the student learning scenario, and the target intelligent agent also includes an emotion recognition module; After determining the exercise correction result according to the user's answer to the after-class exercise and the reference answer to the after-class exercise, the method further includes: Collecting the user's audio and video information during the answer input operation by the emotion recognition module, wherein the audio and video information includes voice input information and image input information; Determine a first emotion recognition result according to the intonation features and the speech speed features in the voice input information, and determine a second emotion recognition result according to the facial expression feature vector and the body movement timing features in the image input information; The first emotion recognition result and the second emotion recognition result are fused according to a preset fusion algorithm to generate a current emotion state score, wherein the current emotion state score is used to characterize the degree of emotion tendency of the user in the positive dimension, the neutral dimension and the negative dimension in the current state; and If the negative dimension score corresponding to the negative dimension exceeds a first score threshold, calling a preset motivational corpus in the local knowledge base to generate and display personalized encouragement information, and reducing the difficulty level of the after-class exercises displayed next time; or, If the negative dimension score is detected to exceed a second score threshold for a preset number of consecutive warning times, the device information of the terminal device of the guardian of the user is retrieved through the third-party plug-in, and the second score threshold is greater than the first score threshold; Determine an emotion fluctuation curve according to the current emotion state score corresponding to the number of warnings, and determine learning state information according to the exercise correction results corresponding to the number of warnings; A warning report is sent to the terminal device of the guardian according to the device information, wherein the warning report carries the emotion fluctuation curve and the learning status information.

7. The method according to claim 4, characterized in that The target teaching scene is the teacher teaching scene, the target application function is a classroom auxiliary function, the target intelligent agent further includes an emotion recognition module, and the intelligent teaching system further includes a classroom terminal in the classroom corresponding to the teacher teaching scene; after the target intelligent agent receives the target response result returned by the large model base, the target application function is realized according to the target response result, including: According to the target response result, determine the classroom recording video collected in real time for the teacher's teaching scene; Performing a video analysis operation based on the classroom recording video to determine the facial expression data corresponding to the student and the body movement data corresponding to the teacher; Determine a class attention score according to the facial expression data, and determine a teacher teaching stress value according to the body movement data, wherein the class attention score is used to represent an average value of the concentration of the target class recorded by the classroom recording video, and the teacher teaching stress value is used to represent the tension of the user in the current teacher teaching scene; and If the duration of the class attention score being lower than the third score threshold is longer than a preset duration, the target agent calls the third-party plug-in to push in-class test questions or group discussion tasks to the classroom terminal, and the in-class test questions and the group discussion tasks are associated with the target subject corresponding to the current teacher teaching scene; or If the teacher's teaching pressure value is greater than or equal to a fourth score threshold, the local knowledge base is called through the target agent to match the teaching design case of the target subject, and the classroom lesson plan of the current teacher's teaching scenario uploaded in advance by the user is obtained; According to the teaching design case, the current classroom lesson plan is updated so that the teaching difficulty level of the updated classroom lesson plan is lower than the teaching difficulty level of the classroom lesson plan before the update.

8. A data processing device based on a multimodal agent, characterized in that: A terminal device applied to an intelligent teaching system, wherein the intelligent teaching system further includes a server; the device includes: An information acquisition unit, used for acquiring a target teaching identity of a user and a target teaching scenario of the user, wherein the target teaching identity is used to indicate that the teaching identity corresponding to the user is a teacher identity or a student identity, and the target teaching scenario is any one of a plurality of reference teaching scenarios matching the teacher identity or the student identity; An agent calling unit, used to determine and call a reference agent corresponding to the target teaching scene as a target agent, each of the reference agents corresponds to one reference teaching scene, the reference agent is used to interact with the user to determine a plurality of preset application functions required to be used in the corresponding reference teaching scene, and the reference agent is communicatively connected with a large model base pre-set in the server; a data interaction unit, configured to collect user request information input by the user through the target agent, and determine a target response result corresponding to a target application function based on a data interaction operation between the target agent and the large model base, wherein the target application function is determined by the user request information and is any one of the plurality of preset application functions; The function realization unit is used to realize the target application function according to the target response result after the target agent receives the target response result returned by the large model base.

9. A terminal device, characterized in that: The method comprises a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 7.

Images