Chinese teaching interaction method and device and storage medium

Abstract

The embodiment of the invention provides a Chinese teaching interaction method and device and a storage medium. In the Chinese teaching method, a coding vector of a current-round student interaction text and a fusion vector of at least one kind of context information corresponding to a plurality of interaction rounds can be input into a language generation model to obtain current-round teaching interaction content. In the implementation mode, the language generation model can accurately analyze the student semantics of the current interaction and the context semantics of the current interaction based on the current student interaction text input by the current interaction round and the context information of the plurality of interaction rounds; and the teaching interaction content of the round matched with the student semantics and the context semantics of the interaction of the round is generated, so that a plurality of rounds of coherent dialogues based on the language environment are realized. In a Chinese teaching scene, the teaching interaction content of the round output by the language generation model can be naturally connected with the interaction content of the historical round up and down, so that the coherence of Chinese teaching is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a Chinese language teaching interactive method, device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, more and more teaching solutions based on artificial intelligence are widely used in various teaching scenarios. For example, in some teaching scenarios, the user's writing ability can be improved through artificial intelligence models. In some typical existing solutions, the artificial intelligence model can generate an article that meets the user's relevant requirements based on the given composition topic and related requirements through a large language generation model and related technologies. The generated article can be provided to the user for reference, thereby guiding the user's writing. In other typical existing solutions, the artificial intelligence model can obtain the composition completed by the user according to a specific composition topic and related requirements, and score the composition completed by the user according to the composition topic and related requirements. Based on the scoring, comments and modifications to the composition can also be generated.

[0003] However, the above method is only a presentation of the generated results, and cannot provide effective feedback and guidance to users during the teaching process. Therefore, a new solution is needed. Summary of the invention

[0004] Multiple aspects of the present application provide a Chinese teaching interaction method, device and storage medium for conducting multiple rounds of dialogue interaction with users during the teaching process, thereby further approaching the real teaching scene.

[0005] The embodiment of the present application provides a Chinese teaching interaction method, comprising: in a current interaction round, obtaining a student interaction text for this round; the student interaction text for this round includes at least: the user's reply content to the content of the previous round of teaching interaction; encoding the student interaction text for this round to obtain an encoding vector of the student interaction text for this round; obtaining a fusion vector of at least one context information corresponding to multiple interaction rounds; inputting the encoding vector of the student interaction text for this round and the fusion vector of the at least one context information into a language generation model to obtain the content of this round of teaching interaction; and outputting the content of this round of teaching interaction.

[0006] Optionally, obtaining a fusion vector of at least one kind of context information corresponding to multiple interaction rounds includes: obtaining at least one kind of context information corresponding to the multiple interaction rounds; performing text splicing on the at least one kind of context information to obtain a spliced ​​text; encoding the spliced ​​text to obtain an encoding vector of the at least one kind of context information; performing feature extraction on the encoding vector of the spliced ​​text to obtain a feature vector of the at least one kind of context information; and fusing the encoding vector and feature vector of the at least one kind of context information to obtain the fusion vector.

[0007] Optionally, the at least one context information includes: teaching tasks, teaching materials, student interaction texts in historical interaction rounds, teaching interaction content in historical interaction rounds, content analysis results of student interaction texts in this round, and at least one of content analysis results of student interaction texts in historical interaction rounds.

[0008] Optionally, it also includes: performing at least one of the following operations on the student interaction text of this round to obtain the content analysis result of the student interaction text of this round: using the intention recognition model to perform intent recognition on the student interaction text of this round to obtain the intention recognition result; using the emotion analysis model to perform emotion analysis on the student interaction text of this round to obtain the emotion analysis result; using the security detection model to perform security detection on the student interaction text of this round to obtain the security detection result.

[0009] Optionally, it also includes: determining a target image to be displayed based on teaching content and teaching materials; performing visual encoding and feature extraction operations on the target image to obtain a visual feature vector; inputting the encoding vector of the current round of student interaction text and the fusion vector of the at least one context information into a language generation model to obtain the current round of teaching interaction content, including: inputting the visual feature vector, the encoding vector of the current round of student interaction text and the fusion vector of the at least one context information into the language generation model together to obtain the current round of teaching interaction content output by the language generation model.

[0010] Optionally, the target image to be displayed is determined based on the teaching content and teaching materials, including: when the teaching content includes picture information, judging whether there is a corresponding target image in the teaching materials; if yes, obtaining the target image from the teaching materials; if not, using a picture generator to generate the target image corresponding to the teaching content.

[0011] Optionally, the interactive content of this round of teaching includes: picture semantics to be activated; the implementation method also includes: determining a target image corresponding to the teaching content; extracting the picture semantics from the interactive content of this round of teaching; using an entity positioning algorithm to determine a target area in the target image where an entity object corresponding to the picture semantics is located; segmenting the entity object corresponding to the picture semantics from the target area, and rendering the entity object when outputting the interactive content of this round of teaching.

[0012] Optionally, outputting the interactive content of this round of teaching includes: generating a reply text based on the interactive content of this round of teaching, and displaying the reply text; and / or generating audio data based on the interactive content of this round of teaching, and playing the audio data; and / or generating video data based on the interactive content of this round of teaching, and playing the video data.

[0013] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method provided in the embodiment of the present application.

[0014] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, which can implement the steps in the method provided in the embodiment of the present application when the computer program is executed by a processor.

[0015] In the Chinese teaching interaction method provided by the embodiment of the present application, after obtaining the student interaction text of this round, the student interaction text of this round can be encoded to obtain the encoding vector of the student interaction text of this round. In addition, a fusion vector of at least one context information corresponding to multiple interaction rounds can be obtained, and the multiple interaction rounds include historical interaction rounds before the current interaction round. The encoding vector of the student interaction text of this round and the fusion vector of the at least one context information are input into the language generation model to obtain the interactive content of this round of teaching. In this embodiment, based on the student interaction text of this round input in the current interaction round and the context information of multiple interaction rounds, the language generation model can accurately analyze the student semantics of this round of interaction and the context semantics of this round of interaction, and generate the interactive content of this round of teaching that matches the student semantics and context semantics of this round of interaction, thereby realizing a coherent multi-round dialogue based on the language environment. In the Chinese teaching scenario, the interactive content of this round of teaching output by the language generation model can be naturally connected with the interactive content of the historical rounds, thereby improving the coherence of Chinese teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A flowchart of a Chinese teaching interaction method provided by an exemplary embodiment of the present application;

[0018] Figure 2 A schematic diagram of the structure of an online teaching system provided for an exemplary embodiment of the present application;

[0019] Figure 3 A flowchart of a Chinese teaching interaction method provided by another exemplary embodiment of the present application;

[0020] Figure 4 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0023] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0024] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0025] In view of the technical problem that the prior art is unable to conduct multiple rounds of dialogue interactions with users during the process of Chinese teaching interaction, a solution is provided in some embodiments of the present application. The technical solutions provided by the embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0026] Figure 1 is a flowchart of a Chinese teaching interactive method provided by an exemplary embodiment of the present application, and the method may include: Figure 1 Steps shown:

[0027] Step 101: In the current interaction round, obtain the student interaction text of this round; the student interaction text of this round at least includes: the user's reply content to the content of the previous round of teaching interaction.

[0028] Step 102: Encode the student interaction text of this round to obtain the encoding vector of the student interaction text of this round.

[0029] Step 103: Obtain a fusion vector of at least one kind of context information corresponding to multiple interaction rounds.

[0030] Step 104: Input the encoding vector of the student interaction text of this round and the fusion vector of the at least one context information into the language generation model to obtain the content of this round of teaching interaction.

[0031] Step 105: output the interactive content of this round of teaching.

[0032] This application is applicable to Chinese teaching scenarios. In Chinese teaching scenarios, the online teaching system can simulate the teacher's teaching operations in real teaching scenarios, and student users can interact with the online teaching system through terminal devices for multiple rounds, thereby realizing interactive learning without a teacher. Among them, the online teaching system can run independently on the terminal device, or can run in a system composed of a terminal device and a server, which is not limited in this embodiment.

[0033] The current interaction round may be any non-first-round interaction round. In the non-first-round interaction round, the student interaction text of this round may be obtained. The user's student interaction text of this round may be obtained from the interaction information input by the user in the current interaction round. The interaction information input by the user may include at least one of text information, voice information and picture information. When the interaction information input by the user includes voice information, the voice information input by the user may be converted into text information by ASR (Automatic Speech Recognition, speech recognition technology) to obtain the student interaction text of this round. When the interaction information input by the user includes picture information, the picture information input by the user may be converted into text information by OCR (Optical Character Recognition, optical character recognition) to obtain the student interaction text of this round. When the interaction information input by the user includes multimodal interaction information, the interaction information of the non-text modality may be converted into text information and merged with the text information input by the user to obtain the student interaction text of this round. For example, when the interactive information input by the user includes voice information and text information, the voice information can be converted into text information and then merged with the text information input by the user to obtain the student interaction text of this round. For example, in the current interaction round, the user inputs "This is a short essay I wrote about my mother" by voice, and the user inputs the short essay content "My favorite mother..." by text. After performing voice recognition on the above voice content, the recognition result is merged with the text input content to obtain the student interaction text of this round corresponding to "This is a short essay I wrote about my mother, my favorite mother..."

[0034] When there are multiple rounds of interaction, there is a certain connection between adjacent dialogue rounds in the semantic context, and the student interaction text of this round may at least include: the user's reply content to the previous round of teaching interaction content. For example, in the previous round of interaction, the teaching interaction content output by the online teaching system is: "What season does the author describe the story taking place?". The user's reply content to the previous round of teaching interaction content is: "Autumn". In other optional embodiments, the student interaction text of this round may also include: new interaction text input by the user in the current interaction round. For example, in the previous round of interaction, the teaching interaction content output by the online teaching system is: "What season does the author describe the story taking place?". The user's reply content to the previous round of teaching interaction content is: "Autumn (i.e., the reply content); What are the poems describing autumn? (i.e., the new interaction text)".

[0035] After obtaining the student interaction text of this round, the student interaction text of this round can be encoded to obtain the encoding vector of the student interaction text of this round. The text is encoded to map the text to the vector space to obtain a vector of fixed dimension, so as to facilitate the recognition of text semantics in the vector space.

[0036] In this embodiment, a fusion vector of at least one kind of context information corresponding to multiple interaction rounds may be obtained, and the encoding vector of the student interaction text of this round and the fusion vector of the at least one kind of context information may be input into the language generation model.

[0037] Among them, multiple interaction rounds include some or all historical interaction rounds before the current interaction round. In some embodiments, in addition to historical interaction rounds, multiple interaction rounds may also include the current interaction round. Among them, the context information is used to describe the preface and previous content of the dialogue process in the Chinese teaching scenario. Optionally, the context information of multiple interaction rounds may include: at least one of: teaching tasks, teaching materials, student interaction texts in historical interaction rounds, teaching interaction content in historical interaction rounds, content analysis results of student interaction texts in this round, and content analysis results of student interaction texts in historical interaction rounds. This embodiment includes but is not limited to this.

[0038] Among them, teaching tasks are used to describe relevant teaching information such as teaching objectives and teaching points. For example, teaching tasks can be described in the following format: {"teaching objectives": teach students to simply describe outdoor scenery, "teaching points": the order of description from far to near, ...}. Among them, teaching materials refer to materials required for teaching, such as picture materials, audio materials, etc. For example, in the teaching scenario of picture writing, teaching materials may include given pictures.

[0039] The content analysis results of the student interaction text in any interaction round include at least one of the following: intent recognition results, sentiment analysis results, and security detection results. The intent recognition results can guide the language generation model to generate corresponding reply texts according to the user's interaction intentions, so as to generate reply texts under specific interaction intentions. For example, when teaching reading comprehension of the same article, the interaction intentions of different users can be: understanding the author's life, learning rhetoric, and learning the storyline. Under the guidance of the above different interaction intentions, the language generation model can generate different reply texts. The sentiment analysis results can guide the language model to generate corresponding reply texts according to the user's emotions, so as to provide different interactive content according to the emotions of different users. For example, during the teaching process, if it is recognized that the user is in a bored mood, a question-type reply content can be generated to stimulate the user's interest in learning and guide the user to continue learning. The security detection results are used to determine whether the student interaction text complies with the security rules to create a good learning atmosphere. If the security detection result of the student interaction text is unsafe, the user can be prompted to re-enter.

[0040] In each interaction round, after obtaining the student interaction text of this round, the content analysis module can be used to perform content analysis on the student interaction text. Among them, the content analysis module can be implemented as a text classification model, such as a text classification model based on BERT (Bidirectional Encoder Representation from Transformers). The content analysis module may include multiple text classification models, such as an intent recognition model, a sentiment analysis model, and a security detection model. In each interaction round, after the student interaction text is input into the content analysis module, the student interaction text can be processed by the intent recognition model, the sentiment analysis model, and the security detection model, respectively, and the intent recognition result, the sentiment analysis result, and the security detection result are output respectively. For example, taking the current round of interaction as an example, after obtaining the student interaction text of this round, at least one of the following operations can be performed on the student interaction text of this round to obtain the content analysis result of the student interaction text of this round: use the intention recognition model to perform intent recognition on the student interaction text of this round to obtain the intention recognition result; use the sentiment analysis model to perform sentiment analysis on the student interaction text of this round to obtain the sentiment analysis result; use the security detection model to perform security detection on the student interaction text of this round to obtain the security detection result.

[0041] The above content analysis results can be stored in a specific format and applied to the subsequent dialogue generation process to achieve the purpose of guiding dialogue generation. For example, in some embodiments, the content analysis results can be stored in the following format: {"Intent Recognition": Chatting, "Sentiment Analysis": Neutral, "Safety Detection": Compliance}.

[0042] In some optional embodiments, when obtaining a fusion vector of at least one context information corresponding to multiple interaction rounds, at least one context information corresponding to the multiple interaction rounds may be obtained, and the at least one context information may be text-concatenated to obtain a concatenated text. After obtaining the concatenated text, a text encoder may be used to encode the concatenated text to obtain an encoding vector of the at least one context information, and a feature extraction model may be used to extract features from the encoding vector of the concatenated text to obtain a feature vector of the at least one context information. The text encoder may include but is not limited to Transformer (a transformation model based on a self-attention mechanism), CNN (Convolutional Neural Networks), etc. The feature extraction model may include a text feature extraction model based on deep learning.

[0043] After obtaining the encoding vector and feature vector of the at least one context information, the encoding vector and feature vector of the at least one context information can be fused to obtain a fusion vector. Wherein, the way of fusing the encoding vector and feature vector of the at least one context information may include: splicing the encoding vector of the at least one context information and the feature vector of the at least one context information. Thus, the encoding vector and feature vector of the context information can be used as inputs of the language generation model and contribute to the generation process of the teaching interaction content. The language generation model can calculate the input data based on the pre-trained model parameters and output the content of this round of teaching interaction. Wherein, the teaching interaction content is used to simulate the teaching content output by the teacher in the real teaching scene according to the input content of the students. In the process of calculating the input data, the language generation model can analyze the semantics of this round of interaction and the contextual semantics of this round of interaction according to the current student interaction text input in the current interaction round and the contextual information of multiple interaction rounds, so as to facilitate the realization of multi-round dialogue teaching in the context language environment and enable multiple rounds of dialogue to be naturally connected.

[0044] Among them, the language generation model can be LLaVA (Large Language and Vision Assistant), which is not limited in this embodiment. After obtaining the interactive content of this round of teaching, the interactive content of this round of teaching can be output. Among them, the interactive content of this round of teaching can be output in a multimodal output mode, and the multimodal output mode may include a text output mode, an image output mode, an audio output mode, a video output mode, etc., which is not limited in this embodiment.

[0045] In some optional embodiments, a reply text may be generated based on the interactive content of this round of teaching (for example, the reply text is extracted from the interactive content of this round of teaching), and the reply text is displayed. Optionally, audio data may be generated based on the interactive content of this round of teaching, and the audio data may be played. For example, the online teaching system may convert the extracted reply text into audio data, and play the audio data. Optionally, video data may be generated based on the interactive content of this round of teaching, and the video data may be played. For example, the online teaching system may generate video data based on the reply text, and play the video data. Optionally, the online teaching system may also play the converted audio data while displaying the reply text. Optionally, the online teaching system may also play the converted video data while displaying the reply text. Among them, the method of converting the reply text into audio data may include: using a text-to-speech (TTS) algorithm to convert the reply text into speech. Among them, the method of generating video data based on the reply text may include: using keywords in the reply text to perform regular matching on a preset video material library to select an adapted video template, animation effect, and background music from the preset video material library. According to the selected video template, animation effect and background music, video data can be generated in combination. Based on this implementation method, through voice interaction, the interaction threshold with users can be further lowered, and teaching efficiency can be improved.

[0046] In this embodiment, after obtaining the student interaction text of this round, the student interaction text of this round can be encoded to obtain the encoding vector of the student interaction text of this round. In addition, a fusion vector of at least one context information corresponding to multiple interaction rounds can be obtained, and the multiple interaction rounds include historical interaction rounds before the current interaction round. The encoding vector of the student interaction text of this round and the fusion vector of the at least one context information are input into the language generation model to obtain the content of the current round of teaching interaction. In this embodiment, based on the student interaction text of this round input in the current interaction round and the context information of multiple interaction rounds, the language generation model can accurately analyze the student semantics of this round of interaction and the context semantics of this round of interaction, and generate the current round of teaching interaction content that matches the student semantics and context semantics of this round of interaction, thereby realizing a coherent multi-round dialogue based on the language environment. In the Chinese teaching scenario, the content of the current round of teaching interaction output by the language generation model can be naturally connected with the content of the interaction of the historical rounds, thereby improving the coherence of Chinese teaching.

[0047] In the interactive process of Chinese teaching, the online teaching system can also output visual information to increase the richness of teaching content from multiple sensory perspectives. An example is given below.

[0048] In some optional embodiments, the online teaching system may determine the target image to be displayed based on the teaching content and teaching materials. The teaching content and teaching materials may be determined by the online teaching system based on the user's learning needs. For example, when a user starts a reading comprehension learning course for a certain article, the teaching content is the reading comprehension of the article, and the teaching materials are the pictures, audio and other teaching materials corresponding to the article. For another example, when a user starts a picture-based writing learning course for a certain image, the teaching content is writing teaching for the image, and the teaching materials are the image and the corresponding writing model.

[0049] Optionally, when the teaching content includes picture information, it can be determined whether there is a corresponding target image in the teaching material. The target image can be a picture, an animation or a video, etc. If there is a corresponding target image in the teaching material, the target image is obtained from the teaching material; if there is no corresponding target image in the teaching material, the target image is generated using a picture generator. The picture generator can be a text-to-image generation model based on Stable-Diffusion (a text-to-image generation model based on latent diffusion models). The picture generator can generate corresponding images based on the text information in the teaching content. For example, when the teaching content contains the text "pine tree", the picture generator can generate a pine tree picture based on the teaching content. After determining the target image based on the above-mentioned implementation method, the target image can be visually encoded and feature extracted to obtain a visual feature vector. When the target image is a video, the video can be framed at a fixed frequency to obtain the picture contained in the video, and visual encoding and feature extraction operations can be performed on the picture. After the visual feature vector is obtained, the visual feature vector, the encoding vector of the fusion vector of the student interaction text in this round and at least one context information can be input into the language generation model. That is, the input of the language generation model includes: the visual feature vector, the encoding vector of the student interaction text in this round and the fusion vector of the at least one context information.

[0050] In some optional embodiments, the interactive content of this round of teaching outputted by the language processing model may include: picture semantics to be activated, wherein the picture semantics is used to be activated when outputting the interactive content of this round of teaching to display the target image.

[0051] In this embodiment, the language generation model can calculate based on the input visual feature vector and the current round of student interaction text, and output the picture semantics to be activated. That is, the output of the language generation model may include: the content of the current round of teaching interaction and the picture semantics to be activated. Optionally, the language generation model can output the content of the current round of teaching interaction and the picture semantics to be activated in a structured information description method, and the structured information description method can be: {"reply text": "xxx", "picture semantics": "xxx"}. For example, {"reply text": "Children, look at the lamb in the picture", "picture semantics": "lamb"}.

[0052] When outputting the interactive content of this round of teaching, the online teaching system can display the target image according to the semantics of the image, thereby outputting the teaching results of this round to the user from multiple dimensions.

[0053] Optionally, when outputting the interactive content of this round of teaching, the entity objects in the target image can be displayed. The entity objects refer to specific objects in the image, such as people, animals, plants or other objects in the image. Optionally, in any current interactive round, the image semantics to be activated can be extracted from the interactive content of this round of teaching output by the language generation module, and the target area where the entity object corresponding to the image semantics is located can be determined in the target image by using the entity positioning algorithm. After determining the target area, the entity object corresponding to the image semantics can be segmented from the target area, and the entity object can be rendered when outputting the interactive content of this round of teaching.

[0054] Among them, the entity positioning algorithm is used to predict the position of the entity object corresponding to the prompt word in the image according to the prompt word. In some embodiments, the entity positioning algorithm can be a segmentation algorithm (Contrastive Language-Image Pre-training Segmentation, CLIPSeg) algorithm based on a language-image pre-training model of contrastive learning. The CLIPSeg algorithm can segment the specified content in the image according to the input prompt word. In this embodiment, the picture semantics of the target image can be input as a prompt word into the CLIPSeg model, and the target area output by the CLIPSeg model can be obtained. For example, the picture semantics of the target image is "lamb", and after {picture semantics: "lamb"} is input into the CLIPSeg model, the position area of ​​the "lamb" in the target image output by the CLIPSeg model can be obtained. When displaying the target image, the "lamb" is segmented from the target image according to the position area where the "lamb" is located, and the local image corresponding to the "lamb" is rendered.

[0055] In some optional embodiments, during the rendering of the entity object, an animation effect of the entity object may be further displayed to increase the flexibility of the visual dimension. The animation effect may be shaking, flashing, size change, etc., thereby optimizing the display effect of the teaching interaction information by combining movement and stillness.

[0056] The following will be combined Figure 2 , the interactive method for Chinese teaching provided in the embodiment of the present application is further illustrated. Figure 2 As shown, the online teaching system for executing the Chinese teaching interactive method may include the following modules: ASR module 201, text encoder 202, content analysis module 203, image generator 204, visual encoder 205, visual feature extraction model 206, multi-round dialogue manager 207, text feature extraction model 208, language generation model 209, TTS module 210 and entity positioning module 211.

[0057] like Figure 3 As shown, in the current interaction round, the online teaching system can receive user input, and the user input includes voice input and / or text input. When the user input includes voice input and text input, the voice input can be converted into text input by the ASR module 201 and merged with the text input. After obtaining the text input of the current interaction round, the text information is encoded using the text encoder 202 to obtain the text encoding vector of the current round of student interaction.

[0058] At the same time, if Figure 3 As shown, the text information can be analyzed by the content analysis module 203. The content analysis module 203 mainly includes algorithm models related to intent recognition, sentiment analysis and security detection. The underlying model of the above algorithm model can be a text classification model based on BERT. After the content analysis module 203 obtains the content analysis result of the text information, it can format the content analysis result and input the organized content analysis result into the multi-round dialogue manager 207 for information storage, so as to achieve the purpose of guiding the subsequent dialogue generation.

[0059] like Figure 3As shown, the online teaching system can determine whether there is a need to generate pictures based on the teaching content and teaching materials. If the teaching content contains a description of the picture and the teaching material does not have relevant picture information, it can be determined to generate the relevant picture. If the teaching content contains a description of the picture and the teaching material has relevant visual information, there is no need to execute the picture generation process, and the visual information in the teaching content and teaching materials can be directly used. Among them, the picture generation operation is performed by the picture generator 204. The input of the picture generator 204 is the relevant picture description in the teaching content, and the output is the corresponding image. The generated image and the picture information in the teaching material can enter the visual information extraction stage as a visual information stream. Figure 3 As shown, in the visual information extraction stage, the visual encoder 205 can be used to encode the input visual information stream to obtain a visual coding vector, and the visual feature extraction model 206 can be used to extract features from the visual coding vector to obtain a visual feature vector. Among them, the visual information stream may include but is not limited to the pictures, animations, videos and other related information used in this teaching. When extracting visual information from video information, frame extraction processing can be performed at a fixed frequency to obtain the corresponding picture, and the visual information extraction operation can be performed on the picture frame by frame.

[0060] The multi-round dialogue manager 207 can manage the context information of multiple interaction rounds. The context information includes: teaching tasks, teaching materials, student interaction texts in historical interaction rounds, teaching interaction content in historical interaction rounds, and the content analysis results of each round obtained from the content analysis module 203. Figure 3 As shown, the text encoder 202 can be used to perform text encoding on each of the above context information to obtain a text encoding vector of the context information, and the text feature extraction model 208 can be used to perform feature extraction on the text encoding vector to obtain a text feature vector of the context information.

[0061] like Figure 3 As shown, the text encoding vector of the current interaction round output by the text encoder 202, the text encoding vector of the context information of multiple interaction rounds output by the text encoder 202, and the text feature vector of the context information of multiple interaction rounds output by the text feature extraction model 208 can be input into the language generation model 209. The language generation model 209 can use the pre-trained language generation ability to output the content of the current round of teaching interaction to simulate the teacher's Chinese teaching. Among them, the teaching interaction content output by the language generation model 209 includes the reply text and the picture semantics to be activated.

[0062] like Figure 3 As shown, the reply text output by the language generation model 209 can enter the multi-round dialogue manager 207 and be merged with the context information managed in the multi-round dialogue manager 207 to be applied to the next dialogue round. Figure 3As shown, the reply text output by the language generation model 209 can be directly output as text. Figure 3 As shown, the reply text output by the language generation model 209 can also be synthesized by the TTS module 210 to obtain the audio output to the user and perform voice output.

[0063] like Figure 3 As shown, the picture semantics output by the language generation model 209 can be input into the entity positioning module 211. The entity positioning module 211 can determine the position of the entity to be activated in the picture to be displayed based on the CLIPSeg algorithm, and feed back the position information to the image to be displayed. Furthermore, the online teaching system can animate the corresponding position area and output the image. Based on this implementation, when outputting teaching information, the entity objects corresponding to the semantics can be dynamically displayed. On the one hand, the teaching process can be visualized and concretized. On the other hand, the image of the entity object can be used as a reward mechanism to reward the user's performance, thereby stimulating the user's interest in learning.

[0064] Those skilled in the art should understand that Figure 2 Each module of the online teaching system can be executed accordingly Figure 1 The details of the steps in the method are not repeated here. For details, please refer to Figure 1 Described in the Methods section.

[0065] The following will take the teaching scenario of Chinese reading comprehension as an example to illustrate the multiple rounds of interactions in the teaching process.

[0066] For the purpose of teaching Chinese reading comprehension, the online teaching system can show users an article containing pictures. In the first interactive round, the student interactive text is the text part of the article, the visual information stream is the picture in the article, and the context information is initialized as the teaching task and teaching material of reading comprehension. After the text encoding vector of the context information of the first interactive round, the text feature vector of the context information, and the visual feature vector of the visual information stream are fused, they can be input into the language generation model together with the text encoding vector of the student interactive text of the first interactive round. The language generation model can output the following reply text: "Little friend, let's take a look at this article and pay attention to the information in the picture..." The online teaching system can convert the above reply text into audio information and play it to the user.

[0067] In the subsequent interaction process, the user's input and the output of the language generation model can be accumulated into the context information of multiple interaction rounds to guide the subsequent interaction process, thereby effectively achieving the purpose of multiple rounds of dialogue. For example, in the second interaction round, the output in the online teaching system may include: "What rhetorical devices does the author use to describe spring?" The student interaction text input by the user may be: "Metaphors and personification are used". In the third interaction round, the output in the online teaching system may include: "Great! In fact, the author also uses exaggerated expressions, such as the first sentence of the second paragraph. Do you see the little swallow in the picture?" The student interaction text input by the user may be: "Yeah, I see the little swallow". In the process of the third round of dialogue, the little swallow in the picture can display a flashing effect under the activation of semantics to increase the user's learning experience. Then, through multiple rounds of interaction with the user, the effective reading comprehension teaching purpose is achieved.

[0068] In addition to the language teaching interactive method described in the above embodiment, the present application embodiment also provides a model training method, which is used to Figure 2 The following will continue to combine Figure 2 An exemplary description is given.

[0069] The training process may include at least a submodule training phase and a joint training phase. The submodule training phase is used to train the modules that can be trained independently in the online teaching system. The modules that can be trained independently may include Figure 2 The picture generator 204 and the language generation model 209 shown. In the submodule training stage, the above modules can be initialized separately, and the corresponding training data can be used to train the above multiple modules. Among them, the training data of each independent module may include manually annotated sample data. For example, the input of the picture generator 204 can be a text-image pair. The input of the language generation model 209 can be a text pair, which can include: dialogue input text, previous context text and real dialogue output text.

[0070] After the initial training of the above modules is completed, the joint training stage can be entered to jointly train multiple models in the online teaching system. In the joint training stage, the training data can be a data group, which includes: dialogue input text, previous context text, teaching content, teaching materials, real entity location and real dialogue output text. The overall output of the online teaching system is the predicted text and the predicted entity location. Based on the gap between the predicted text and the real dialogue output text, and the gap between the predicted entity location and the real entity location, the joint training loss can be determined. Based on the joint training loss, the gradient return mechanism can be used to fine-tune the model parameters of one or more modules in the online teaching system until the training loss converges to the specified range, stop the gradient descent, and obtain the above modules that have been trained. After the training is completed, the online teaching system can be put into use to simulate and supplement the real teaching scene.

[0071] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0072] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0074] Figure 4 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application is shown. Figure 4As shown, the electronic device includes: a memory 401 , a processor 402 and a communication component 403 .

[0075] The memory 401 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.

[0076] The processor 402 is coupled to the memory 401 and is used to execute the computer program in the memory 401, so as to: obtain the student interaction text of this round in the current interaction round; the student interaction text of this round includes at least: the user's reply content to the content of the previous round of teaching interaction; encode the student interaction text of this round to obtain the encoding vector of the student interaction text of this round; obtain the fusion vector of at least one context information corresponding to multiple interaction rounds; input the encoding vector of the student interaction text of this round and the fusion vector of the at least one context information into the language generation model to obtain the content of this round of teaching interaction; and output the content of this round of teaching interaction.

[0077] Optionally, when obtaining a fusion vector of at least one kind of context information corresponding to multiple interaction rounds, the processor 402 is specifically used to: obtain the at least one kind of context information corresponding to the multiple interaction rounds; perform text splicing on the at least one kind of context information to obtain a spliced ​​text; encode the spliced ​​text to obtain an encoding vector of the at least one kind of context information; perform feature extraction on the encoding vector of the spliced ​​text to obtain a feature vector of the at least one kind of context information; and fuse the encoding vector and the feature vector of the at least one kind of context information to obtain the fusion vector.

[0078] Optionally, the at least one context information includes: teaching tasks, teaching materials, student interaction texts in historical interaction rounds, teaching interaction content in historical interaction rounds, content analysis results of student interaction texts in this round, and at least one of content analysis results of student interaction texts in historical interaction rounds.

[0079] Optionally, the processor 402 is also used to: perform at least one of the following operations on the current round of student interaction text to obtain a content analysis result of the current round of student interaction text: use an intention recognition model to perform intent recognition on the current round of student interaction text to obtain an intention recognition result; use an emotion analysis model to perform emotion analysis on the current round of student interaction text to obtain an emotion analysis result; use a security detection model to perform security detection on the current round of student interaction text to obtain a security detection result.

[0080] Optionally, the processor 402 is also used to: determine the target image to be displayed based on the teaching content and teaching materials; perform visual encoding and feature extraction operations on the target image to obtain a visual feature vector; input the encoding vector of the current round of student interaction text and the fusion vector of the at least one context information into the language generation model to obtain the content of the current round of teaching interaction, including: inputting the visual feature vector, the encoding vector of the current round of student interaction text and the fusion vector of the at least one context information into the language generation model together to obtain the content of the current round of teaching interaction output by the language generation model.

[0081] Optionally, when the processor 402 determines the target image to be displayed based on the teaching content and teaching materials, it is specifically used to: when the teaching content includes picture information, determine whether there is a corresponding target image in the teaching material; if yes, obtain the target image from the teaching material; if not, use the picture generator to generate the target image corresponding to the teaching content.

[0082] Optionally, Figure 4 The electronic device shown also includes a display component 404. The interactive content of this round of teaching includes: picture semantics to be activated, and the processor 402 is further used to: determine a target image corresponding to the teaching content; extract the picture semantics from the interactive content of this round of teaching; use an entity positioning algorithm to determine a target area where a physical object corresponding to the picture semantics is located in the target image; segment the physical object corresponding to the picture semantics from the target area, and control the display component 404 to render the physical object when outputting the interactive content of this round of teaching.

[0083] Optionally, Figure 4 The electronic device shown also includes an audio component 405. Optionally, when outputting the interactive content of this round of teaching, the processor 402 is specifically used to: generate a reply text according to the interactive content of this round of teaching (for example, extract the reply text from the interactive content of this round of teaching); control the display component 404 to display the reply text; and / or, generate audio data according to the interactive content of this round of teaching, for example, the extracted reply text can be converted into audio data, and the audio component 405 can be controlled to play the audio data; and / or, generate video data according to the interactive content of this round of teaching, for example, video data can be generated according to the reply text, and the display component 404 and the audio component 405 can be controlled to play the video data.

[0084] Further, if Figure 4 As shown, the electronic device also includes: a power supply component 406 and other components. Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 Components shown.

[0085] Among them, the memory 401 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0086] The communication component 403 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as Wi-Fi (wireless network communication technology), 2G (such as Global System for Mobile Communications (GSM)), 3G (such as Wideband Code Division Multiple Access (WCDMA), 4G (such as Long Term Evolution (LTE)), 4G+ (such as upgraded Long Term Evolution (LTE-Advanced, LTE-A)) or 5G (5th Generation Mobile Communication Technology), or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0087] Among them, the display component 404 includes a screen, and the screen may include a liquid crystal display (Liquid Crystal Display, LCD) and a touch panel (Touch panel, TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0088] Among them, the audio component 405 can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0089] The power supply component 406 is used to provide power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing and distributing power to the device where the power supply component is located.

[0090] In this embodiment, after obtaining the student interaction text of this round, the electronic device can encode the student interaction text of this round to obtain the encoding vector of the student interaction text of this round. In addition, a fusion vector of at least one context information corresponding to multiple interaction rounds can be obtained, and the multiple interaction rounds include historical interaction rounds before the current interaction round. The encoding vector of the student interaction text of this round and the fusion vector of the at least one context information are input into the language generation model to obtain the interactive content of this round of teaching. In this embodiment, based on the student interaction text of this round input in the current interaction round and the context information of multiple interaction rounds, the language generation model can accurately analyze the student semantics of this round of interaction and the context semantics of this round of interaction, and generate the interactive content of this round of teaching that matches the student semantics and context semantics of this round of interaction, thereby realizing a coherent multi-round dialogue based on the language environment. In the Chinese teaching scenario, the interactive content of this round of teaching output by the language generation model can be naturally connected with the interactive content of the historical rounds, thereby improving the coherence of Chinese teaching.

[0091] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0092] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM (Compact Disc Read-Only Memory, read-only optical disc read-only memory), optical storage, etc.) containing computer-usable program codes.

[0093] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0097] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0098] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Parallel Random Access Machine (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0100] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A Chinese language teaching interactive method, characterized in that: include: In the current interaction round, obtain the student interaction text of this round; The student interaction text of this round includes at least: the user's reply content to the previous round of teaching interaction content; Encoding the student interaction text of this round to obtain an encoding vector of the student interaction text of this round; Obtaining a fusion vector of at least one kind of context information corresponding to multiple interaction rounds; Inputting the encoding vector of the student interaction text of this round and the fusion vector of the at least one context information into the language generation model to obtain the content of the teaching interaction of this round; Output the interactive content of this round of teaching.

2. The method according to claim 1, characterized in that Obtaining a fusion vector of at least one context information corresponding to multiple interaction rounds, including: Acquire at least one kind of context information corresponding to the multiple interaction rounds; Performing text splicing on the at least one context information to obtain a spliced ​​text; Encoding the concatenated text to obtain an encoding vector of the at least one context information; Performing feature extraction on the encoding vector of the concatenated text to obtain a feature vector of the at least one context information; The encoding vector and the feature vector of the at least one context information are fused to obtain the fused vector.

3. The method according to claim 2, characterized in that The at least one context information includes: at least one of: teaching tasks, teaching materials, student interaction texts in historical interaction rounds, teaching interaction content in historical interaction rounds, content analysis results of student interaction texts in this round, and content analysis results of student interaction texts in historical interaction rounds.

4. The method according to claim 1, characterized in that: Also includes: Perform at least one of the following operations on the student interaction text of this round to obtain the content parsing result of the student interaction text of this round: Using the intention recognition model to perform intention recognition on the student interaction text of this round, and obtain an intention recognition result; Using a sentiment analysis model to perform sentiment analysis on the student interaction texts of this round to obtain sentiment analysis results; The security detection model is used to perform security detection on the student interaction text of this round to obtain a security detection result.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: Determine the target image to be displayed according to the teaching content and teaching materials; Performing visual encoding and feature extraction operations on the target image to obtain a visual feature vector; Inputting the encoding vector of the student interaction text of this round and the fusion vector of the at least one context information into the language generation model to obtain the content of the teaching interaction of this round, including: The visual feature vector, the encoding vector of the student interaction text of this round, and the fusion vector of the at least one context information are input into the language generation model together to obtain the content of the teaching interaction of this round output by the language generation model.

6. The method according to claim 5, characterized in that According to the teaching content and teaching materials, determine the target image to be displayed, including: When the teaching content includes picture information, determining whether there is a corresponding target image in the teaching material; If yes, obtaining the target image from the teaching material; If not, a picture generator is used to generate the target image corresponding to the teaching content.

7. The method according to claim 1, characterized in that The interactive content of this round of teaching includes: the semantics of the picture to be activated; the method also includes: Determine the target image corresponding to the teaching content; Extracting the image semantics from the interactive content of this round of teaching; Using an entity positioning algorithm, determine in the target image a target area where an entity object corresponding to the semantics of the image is located; The entity object corresponding to the semantics of the image is segmented from the target area, and the entity object is rendered when outputting the interactive content of this round of teaching.

8. The method according to any one of claims 1 to 4, characterized in that: Output the interactive content of this round of teaching, including: Generate a reply text based on the content of the current round of teaching interaction, and display the reply text; and / or, Generate audio data according to the interactive content of the current teaching round, and play the audio data; and / or, Generate video data based on the interactive content of this round of teaching, and play the video data.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to: perform the steps in the method according to any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement the Chinese language teaching interactive method described in any one of claims 1-8.

Images