A system, method and user terminal for assisting in learning a foreign language based on a large model
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2024-01-18
- Publication Date
- 2026-07-24
Smart Images
Figure CN118135855B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a system, method and user terminal for assisted foreign language learning based on a large model. Background Technology
[0002] Foreign language learning is a primary focus for middle school students. However, many students face various challenges and problems in the process, and their learning needs are becoming increasingly personalized. Taking English learning as an example, while there are numerous English learning tools and platforms available, they still fall short in meeting the individualized needs of students. Traditional "one-size-fits-all" learning methods are unlikely to satisfy the needs of all students at this stage. How to enable students to learn English more efficiently through personalized customization is an important research topic.
[0003] Many existing products with similar English learning assistant functions (such as learning tablets) have attempted to utilize personalized learning, but these products often adopt standardized education models, pre-set test banks and explanation methods, making it difficult to develop personalized learning plans based on students' levels.
[0004] In other words, existing technologies, such as learning tablets, offer a poor interactive experience and cannot replace interaction with a real teacher. Furthermore, due to their closed-loop nature, the resources within these devices are limited and cannot cover all learning areas. Summary of the Invention
[0005] This application provides a system, method, and user terminal for assisted foreign language learning based on a large model, aiming to solve the technical problem of how to realize assisted learning products with personalized content, thereby enabling students to learn foreign languages more efficiently.
[0006] In a first aspect, embodiments of this application provide a system for assisted foreign language learning based on a large-scale language model. The system includes a user terminal and a cloud server deploying a large-scale language model, wherein:
[0007] The user terminal is used to acquire the user's pending information, parse the pending information, and obtain target text information;
[0008] The user terminal is also used to respond to user operation information regarding the target text information and send the operation information and the target text information to the cloud server;
[0009] The cloud server is used to take the operation information and the target text information as input information to the language model, so that the language model can generate target information based on the input information, wherein the target information is used to assist users in learning foreign languages.
[0010] Preferably, the user terminal is equipped with the LangChain application development framework;
[0011] The user terminal is also used to interact with the language big model of the cloud server using the LangChain application development framework, so that the cloud server uses the operation information and the target text information as input information for the language big model.
[0012] Optionally, the user terminal is further equipped with a preset application programming interface, and the cloud server has a preset recognition model;
[0013] The user terminal is also used to call the preset application programming interface to read the information to be processed, and to call the preset recognition model of the cloud server to parse the information to be processed to obtain target text information;
[0014] Wherein, when the preset application programming interface is an OCR application programming interface, the preset recognition model is an OCR recognition model; when the preset application programming interface is a voice application programming interface, the preset recognition model is a voice recognition model.
[0015] Secondly, embodiments of this application provide a method for assisted foreign language learning based on a large model, the method being applied to a user terminal, the method comprising:
[0016] Obtain the user's pending information, parse the pending information to obtain the target text information;
[0017] Responding to user actions regarding the target text information;
[0018] The operation information and the target text information are sent to the cloud server, so that the cloud server uses the operation information and the target text information as input information for the language big model, and the language big model generates target information based on the input information. The language big model is deployed on the cloud server, and the target information is used to assist users in learning foreign languages.
[0019] In one example, the information to be processed is a test question image related to foreign language learning;
[0020] The step of obtaining the user's pending information includes:
[0021] Play the first voice prompt and display the corresponding first-level function interface, and obtain the first function command input by the user based on the first-level function interface;
[0022] Display and play the second voice prompt, capture the test question image, and determine the question image selected by the user from the test question image;
[0023] Accordingly, the step of parsing the information to be processed to obtain the target text information includes:
[0024] Image analysis is performed on the test question image and the question image to obtain test question text information and question text information;
[0025] The step of responding to user operation information regarding the target text information includes:
[0026] Responding to a second function command from the user regarding the question text information and / or the test question text information;
[0027] The step of sending the operation information and the target text information to the cloud server, so that the cloud server uses the operation information and the target text information as input information for the language big model, and the language big model generates target information based on the input information, includes:
[0028] Based on the LangChain application development framework, the target prompt word is determined according to the first function instruction and the second function instruction, and the target prompt word is sent to the cloud server;
[0029] The test question text information and the question text information are sent to the cloud server as the target text information, so that the cloud server uses the target prompt words and the target text information as input information for the language big model, and the language big model generates target information to assist users in learning foreign languages based on the input information.
[0030] In one example, the first function instruction includes a question explanation function, and the second function instruction includes at least a question organization function, a question explanation function, and a new question generation function:
[0031] Accordingly, the method further includes:
[0032] Based on the target text information, determine the user's learning record and incorrect answers. Combined with the second function instruction and based on the learning record and incorrect answers, generate at least multiple-choice task, reading comprehension task, and fill-in-the-blank task.
[0033] In one example, it also includes: pre-generating a first LangChain sequence chain, a second LangChain sequence chain, and a third LangChain sequence chain based on the LangChain application development framework, wherein the first LangChain sequence chain corresponds to the question organization function, the second LangChain sequence chain corresponds to the question explanation function, and the third LangChain sequence chain corresponds to the new question generation function;
[0034] Accordingly, the method further includes:
[0035] If the second function instruction represents the question organization function, then the first LangChain sequence chain is invoked to interact with the language big model of the cloud server;
[0036] If the second function instruction represents the question explanation function, then the second LangChain sequence chain is invoked to interact with the language big model of the cloud server;
[0037] If the second function instruction represents a new question function, then the third LangChain sequence chain is invoked to interact with the language big model of the cloud server.
[0038] In one example, the first function instruction includes a memorization function and a foreign language conversation function;
[0039] Accordingly, the method further includes:
[0040] Acquire the voice content sent by the user and convert the voice content into voice-text content;
[0041] When the first function instruction represents the memorization function, the text content is compared with the target text information, and the comparison result is displayed after the user completes the memorization.
[0042] When the first function instruction represents the foreign language dialogue function, the text content and the first function instruction are sent to the cloud server, so that the cloud server uses the text content and the first function instruction as input information for the language big model, and the language big model generates dialogue speech based on the input information.
[0043] Thirdly, the present invention also proposes a user terminal, the user terminal comprising at least a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the second aspect above.
[0044] In one example, the user terminal includes an interactive end and a side end;
[0045] The interactive terminal includes at least a display screen, a camera, and a microphone; the camera is used to capture test question images related to foreign language learning; the display screen is used to display the test question images, a first voice prompt, a second voice prompt, and generated target information; the microphone is used to acquire the voice content emitted by the user, and the microphone is also used to play the dialogue voice generated by the language big model on the cloud server.
[0046] The processor and the memory are deployed at the edge, and the computer program is implemented at least based on the LangChain-based application development framework.
[0047] The beneficial effects of this application are that it enables the customization of foreign language learning content based on the user's own situation and needs, thereby improving the user experience for foreign language learners. Attached Figure Description
[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 This is a block diagram of a system for assisted foreign language learning based on a large model, as provided in an embodiment of this application.
[0050] Figure 2 This is a schematic diagram illustrating an embodiment of the language large model workflow provided in this application;
[0051] Figure 3 This is a flowchart illustrating the large-model-based assisted learning method provided in the embodiments of this application;
[0052] Figure 4 This is an example diagram of the content display on the interactive end of the user terminal in this application embodiment;
[0053] Figure 5 This is a simplified structural diagram of a user terminal provided in this application. Detailed Implementation
[0054] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0055] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0056] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0057] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0058] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0059] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0060] Understandably, there is currently a limited amount of research and tools available for personalized English learning based on large-scale models. To address the issues mentioned in the background, this application provides a system, method, and user terminal for assisted foreign language learning based on large-scale models. This system not only customizes course content according to students' foundation and needs but also allows for flexible interactive dialogue with learners, providing real-time feedback and suggestions to help students better learn foreign language knowledge.
[0061] Example 1
[0062] To address the problems mentioned in the background section, this invention proposes a large-model-based assisted foreign language learning system. The system architecture of this embodiment mainly includes a user terminal and a cloud server deploying a large language model, such as... Figure 1As shown, the user terminal includes an interactive end and an edge end. The interactive end includes at least a display screen, a camera, and a microphone. The edge end includes at least a processor (e.g., an AI chip) and a memory, as well as a computer program stored in the memory and executable on the processor (in this embodiment, the computer program is at least based on the LangChain application development framework). The computer program is deployed with preset application programming interfaces, including an OCR API and a speech API. The cloud server has preset recognition models, including an OCR recognition model and a speech recognition model.
[0063] The user terminal is used to acquire the user's pending information, parse the pending information, and obtain target text information;
[0064] In this embodiment, the information to be processed can be test question images related to foreign language learning;
[0065] In the specific implementation, the user terminal uses the camera to capture test question images, reads the captured test question images through the OCR application programming interface, and calls the cloud-based OCR recognition model to parse the text in the test question images, thereby obtaining the test question text information, and using the obtained test question text information as the target text information.
[0066] It should be noted that OCR recognition technology is used to convert the questions in the test question image into text. Students only need to take a picture of the question and select the relevant type and question number, and the system can recognize the question.
[0067] In addition, in other embodiments, a cloud-based approach using Microsoft Azure OCR can be used to recognize test question images collected locally on the user terminal. Considering that many real-world usage scenarios involve issues such as lighting conditions, the aforementioned cloud-based approach can display more detailed test question images.
[0068] The user terminal is also used to respond to user operation information regarding the target text information and send the operation information and the target text information to the cloud server;
[0069] It should be noted that the operational information described in this application may include two levels of functionality (a first level, and a second level below the first level):
[0070] For example, the functions at the first level can include question explanation, memorization, and foreign language dialogue functions; if the function at the first level is "question explanation", then the functions at the second level below it can further include "question organization", "question explanation" and "new question generation".
[0071] The operation information may include the second function instruction corresponding to the second-level function (for example, when a student selects "problem explanation function" or "new question creation function", the instruction to select the specific function is the second function instruction). Of course, the operation information may also include the first function instruction (in one case, since the system will set the second function instruction to be the next level instruction of the first function by default, the user terminal sends the second function instruction to the cloud, and the cloud can determine the "first function instruction" based on the second function instruction).
[0072] In its implementation, this application uses the LangChain application development framework to build a large language model application on the user terminal, so as to achieve the purpose of interacting with the large language model on the cloud server. The LangChain application development framework can connect the language model to other data sources and interact with the external environment, and the user interface can be developed using the Grado open source library.
[0073] Specifically, LangChain integrates LLM models, vector databases, the interaction layer Prompt, external knowledge, and external tools, enabling the free construction of LLM applications. With LLM, we can develop programs more lightweightly.
[0074] For the "problem explanation function", this embodiment can directly call LangChain's language model interface for dialogue. Since it does not have memory capabilities, it cannot link the "problem explanation function" with the "new question generation function". In order to link the "problem explanation function" with the "new question generation function" (that is, to customize questions according to students' learning situation), this embodiment can use LangChain's memory function to save students' historical problem-solving history (mainly wrong questions).
[0075] In practical implementation, when the first level is the "question explanation function", such as Figure 2 As shown, this embodiment breaks it down into three functions: 1) Question organization function: organize the questions in the captured images into a standard format; 2) Question explanation function: explain the core knowledge points of the question and explain each option; 3) New question generation function: generate a similar question with similar knowledge points.
[0076] This embodiment can implement these three functions using three Langchain sequential chain modules respectively:
[0077] Specifically, on the user terminal side, a first LangChain sequence chain, a second LangChain sequence chain, and a third LangChain sequence chain will be generated in advance based on the LangChain application development framework. The first LangChain sequence chain corresponds to the question organization function, the second LangChain sequence chain corresponds to the question explanation function, and the third LangChain sequence chain corresponds to the new question generation function.
[0078] If the second function instruction represents the question organization function, then the first LangChain sequence chain is invoked to interact with the language big model of the cloud server;
[0079] If the second function instruction represents the question explanation function, then the second LangChain sequence chain is invoked to interact with the language big model of the cloud server;
[0080] If the second function instruction represents a new question function, then the third LangChain sequence chain is invoked to interact with the language big model of the cloud server.
[0081] This embodiment implements these three functions using three Langchain sequential chain modules, which greatly simplifies the implementation of complex applications and makes them more modular: the OCR recognition result and the user-inputted question to be explained serve as the chain's input, and the chain outputs "question explanation content" as the text box content for the question explanation in the interactive interface, and generates new questions as the content of new questions in the interactive interface. By sequentially calling these modules, the generated content of the large language model is effectively controlled, making it easier to debug, maintain, and improve the application.
[0082] The cloud server is used to take the operation information and the target text information as input information to the language model, so that the language model can generate target information based on the input information, wherein the target information is used to assist users in learning foreign languages.
[0083] Understandably, the user terminal uses the LangChain application development framework to interact with the language big model on the cloud server, so that the cloud server uses the operation information and the target text information as input information for the language big model.
[0084] As can be seen from the above, the operation information in this embodiment may include the first function instruction and the second function instruction. The user terminal, based on the LangChain application development framework, determines the target prompt according to the first function instruction and the second function instruction, and sends the target prompt to the cloud server. The target prompt can be used to guide the language model to generate and output the specified text content.
[0085] Furthermore, the cloud server will use the target prompt and the target text information (test question text information) as input information for the language big model, and the language big model will generate target information to assist users in learning foreign languages based on the input information.
[0086] Understandably, taking English learning as an example, while cloud-based large language models possess strong English proficiency, they may still encounter situations where they don't understand the question requirements, answer incorrectly, and require repeated prompts. During the research process, the applicant discovered that prompts significantly impact the accuracy of the large language model's responses. Therefore, this embodiment designs different prompts for the three functions mentioned above (i.e., "question organization function," "question explanation function," and "new question generation function"), and combines CoT (Cooperation of Thought) and contextual learning to improve the cloud-based large language model's ability to organize, answer, and generate questions.
[0087] Understandably, many existing products with similar English learning assistant functions (such as learning tablets) have pre-defined question banks and explanation methods, making it difficult to personalize them according to students' levels. The language model in this application possesses rich English knowledge and, as a generative technology, can simulate human natural language generation and dialogue interaction. This embodiment can employ two strategies—few-sample examples and thought chain prompts—to design and optimize prompt words, enabling the user terminal to correctly organize questions, provide answers and explanations, and generate new questions based on students' learning progress for personalized practice.
[0088] For few-shot examples: Large language models demonstrate impressive zero-shot capabilities, but they still perform poorly on more complex tasks when using zero-shot settings. This implementation uses in-context learning with few-shot examples, allowing the language model to directly understand the task based on a few given examples and generate answers by mimicking those examples.
[0089] For Chain of Reasoning (CoT) hints: For complex tasks, limited sample hints are insufficient to obtain reliable answers, especially for tasks involving detailed problem explanations and generating new problems. While the examples provide basic information about the tasks, the task types involve several more reasoning steps. Therefore, this embodiment introduces Chain of Reasoning (CoT) hints to solve more complex reasoning tasks, breaking down each problem into steps and demonstrating them to the model, requiring it to complete the task step by step, and guiding the model to solve the problem itself before drawing a conclusion. This embodiment can use delimiters to clearly separate the different parts of the hint and requires structured output.
[0090] For question generation based on generative artificial intelligence: Benefiting from the powerful long-text modeling and text generation capabilities of large language models, the AI chip determines the user's learning records and incorrect answers based on the target text information. Combining the second function instruction with these records, the chip generates at least multiple-choice, reading comprehension, and fill-in-the-blank tasks. This enables the generation of various types of learning tasks, such as multiple-choice, fill-in-the-blank, and reading comprehension questions, based on the knowledge points being tested, providing students with a more in-depth, personalized, and interactive learning experience.
[0091] This application provides a large-scale model-based system for assisting foreign language learning. It can not only customize course content according to students' foundation and needs, but also engage in dialogue with learners through flexible interactive methods, providing real-time feedback and suggestions, thereby helping students learn foreign language knowledge better.
[0092] Example 2
[0093] Based on the system framework provided in Embodiment 1 above, Embodiment 2 provides an embodiment of a method for assisted foreign language learning based on a large model. The method is applied to a user terminal and mainly includes the following steps:
[0094] Step A1: Obtain the user's pending information;
[0095] Step A2: Parse the information to be processed to obtain the target text information;
[0096] Step A3: Respond to the user's operation information regarding the target text information;
[0097] Step A4: Send the operation information and the target text information to the cloud server, so that the cloud server uses the operation information and the target text information as input information for the language big model, and the language big model generates target information based on the input information. The language big model is deployed on the cloud server, and the target information is used to assist users in learning foreign languages.
[0098] It is understood that steps A1 to A4 above constitute the basic technical solution of the large-model-based assisted foreign language learning method of this application. The specific implementation method has been described in detail in Embodiment 1 above, and will not be repeated here. The following will introduce the basic beneficial effects of the technical solution of this application through practical application scenarios:
[0099] Specifically, in one application scenario embodiment, such as Figure 3 As shown:
[0100] Step A1, "obtaining the user's pending information," specifically includes:
[0101] S01, play the first voice prompt and display the interface of the corresponding first-level function, and obtain the first function command input by the user based on the interface of the first-level function;
[0102] In the specific implementation, when the user (student) starts the user terminal, the screen of the interactive end of the user terminal displays the startup interface, and the sound module of the user terminal plays the first voice prompt "Learning Assistant is enabled, please select a function". At the same time, the screen of the interactive end displays the interface of the corresponding first-level function (the optional first-level functions include "explaining questions", "memorizing", and "foreign language dialogue"). The user can select from the interface of the first-level functions through the touch screen, or select the specific first-level function through voice interaction. The user's "touch screen selection" or "voice selection" is used as the first function command, such as the user selecting "explaining questions".
[0103] S02, display and play the second voice prompt, capture the test question image, and determine the question image selected by the user in the test question image;
[0104] Understandably, users can select the corresponding question from the test question image in several ways. The first method is for the user to place their finger on the test question image paper, specifically indicating the location of the question to be explained. The user's terminal camera takes a picture of the test question image paper containing the user's finger to identify the selected question image and thus recognize the selected question. The second method is for the camera to first take a picture of the test question image paper, and after the display panel shows the captured test question image, the user can select the corresponding question image from it by touching the screen.
[0105] In this specific implementation, this embodiment uses the first topic selection method described above as an example. The sound module plays a second voice prompt such as: "The topic explanation function has been activated, the camera is about to turn on, please point to the topic to be explained with your finger." At this time, the user places their finger on the topic to be explained. Furthermore, if the topic is relatively long (such as reading comprehension) or relatively short (such as Chinese-to-English word translation), no more than two fingers can be used to trace from the beginning to the end of the topic.
[0106] Furthermore, the camera module lens opens, captures an image of the test questions on the desktop, and automatically captures the image of the question pointed to by the finger, sending it to the display screen of the interactive terminal for display:
[0107] Furthermore, the audio module continues to play voice prompts such as, "Please confirm that the question displayed on the screen is the question that needs to be explained." If the user selects "Confirm," the question image and the question image are sent to the cloud-based large model interface; if "Incorrect" is selected, the above steps are repeated. In addition, users can manually adjust the question capture area via the touchscreen (similar to the screenshot function on a mobile phone).
[0108] Accordingly, step A2, "parse the information to be processed to obtain the target text information," specifically includes:
[0109] S03, perform image analysis on the test question image and the question image to obtain test question text information and question text information;
[0110] In a specific implementation, after receiving the test question image and the question image, the AI chip extracts text using OCR technology;
[0111] Optionally, after receiving the test question image and the question image, the AI chip can extract text using OCR technology and play voice prompts such as "Please select the type of question that needs to be explained." The user can select the question type by touching the screen or by voice interaction, and the display screen will send the user's selection to the AI chip.
[0112] Accordingly, step A3, "responding to user operation information regarding the target text information," specifically includes:
[0113] S04, responding to a second function command from the user regarding the question text information and / or the test question text information;
[0114] In its implementation, after receiving the target text information from the OCR module and the question information from the screen module, the AI chip uses LangChain to interact with the large language model deployed in the cloud. The display screen shows the interface for the second-level functions, including three functions: "Question Organization Function," "Question Explanation Function," and "New Question Generation Function." Users select specific second-level functions by clicking function buttons on the screen or through voice interaction, and the user's selection of functions in the second-level function interface is taken as the "second function command."
[0115] Accordingly, step A4 specifically includes:
[0116] S05, based on the LangChain application development framework, the target prompt word is determined according to the first function instruction and the second function instruction, and the target prompt word is sent to the cloud server; the test question text information and the question text information are sent to the cloud server as the target text information, so that the cloud server uses the target prompt word and the target text information as input information for the language big model, and the language big model generates target information to assist users in learning foreign languages based on the input information.
[0117] The AI chip selects the desired prompt based on the user's "secondary function command," and sends the prompt, question, personalized requirements, and other information to the large language model on the cloud server for calculation. The large language model then generates explanation content and a new question, which is sent to the display screen (e.g., ...). Figure 4 (As shown).
[0118] Furthermore, the cloud server will store users' (students') incorrect answer information in the cloud, building a personalized incorrect answer database. Users can use the "Incorrect Answer Database Practice" function to call the cloud-based large language model to categorize and statistically analyze questions and generate new questions accordingly. The large model will complete tasks based on the prompts, such as compiling students' learning records and incorrect answer situations, analyzing and summarizing frequently missed knowledge points, and generating various types of new questions, such as multiple choice, fill-in-the-blank, and reading comprehension, according to user needs. It can personalize various learning materials based on students' learning records and incorrect answer situations, such as multiple choice, reading comprehension, and fill-in-the-blank learning tasks.
[0119] In addition, in some embodiments, the cloud interface also records the questions and categorizes new questions, and provides a printing function. Users connect their user terminals to printers and print paper versions of the test questions as needed by controlling the interactive interface of their user terminals.
[0120] Furthermore, in another application scenario embodiment, if the user selects either the memorization function or the foreign language dialogue function in the interface of the first-level function, the method further includes:
[0121] S02', acquire the voice content sent by the user, and convert the voice content into voice-text content;
[0122] S0201, when the first function instruction represents the memorization function, the voice text content is compared with the target text information, and the comparison result is displayed after the user completes the memorization;
[0123] S0202, when the first function instruction represents the foreign language dialogue function, the speech text content and the first function instruction are sent to the cloud server, so that the cloud server uses the speech text content and the first function instruction as input information for the language big model, and the language big model generates dialogue speech based on the input information.
[0124] In the specific implementation of the memorization function represented by the first function instruction:
[0125] (1) When a user selects the memorization function via the touchscreen, the system activates the preset memorization function, and the sound module plays a preset prompt: "Memorization function is activated, camera is about to turn on, please point to the paragraph you need to memorize with your finger." At this time, the user places their finger on the paragraph to be memorized. Furthermore, if the memorization paragraph is long, no more than two fingers can be used to trace from the beginning to the end of the question;
[0126] (2) The camera module lens is turned on to capture the image of the desktop test questions and automatically captures the image of the question pointed to by the finger and sends it to the display.
[0127] (3) The audio module plays the prompt: "Please confirm that the question displayed on the screen is the one that needs to be memorized." If the user selects "Confirm," the image will be sent to the cloud; if "Incorrect" is selected, steps 2-5 will be repeated. Furthermore, the user can manually adjust the scope of the question capture box via the touchscreen (refer to the mobile phone screenshot function).
[0128] (4) After receiving the image in the cloud, the text is extracted by OCR and the prompt "Please start memorizing" is played. The microphone starts to capture the user's voice and convert it into text. The text is automatically compared with the aforementioned OCR text and a comparison result is generated.
[0129] (5) After the recitation is completed, the comparison results are displayed on the screen.
[0130] In the specific implementation of the foreign language dialogue function represented by the first function instruction:
[0131] (1) When the user selects the foreign language dialogue function through the touch screen, the system starts the preset foreign language dialogue function and the sound module plays the preset prompt: "Foreign language dialogue function is turned on, microphone function is about to be turned on, please speak into the microphone." At this time, the user speaks a foreign language into the microphone. After the user stops speaking for a few seconds, the user terminal will turn off the microphone function. The AI chip of the user terminal receives the voice content sent by the user and converts the voice content into voice text content.
[0132] (2) In addition, the camera can be turned on to capture the test question image on the desktop after the microphone function is turned off or before it is turned off. The content information of the test question image in this embodiment can determine the dialogue scene text of the dialogue voice content finally generated by the system, and the test question image is converted into dialogue scene text through OCR technology.
[0133] (3) After receiving the voice and text content and the dialogue scenario text, the AI chip on the user terminal uses LangChain to interact with the large language model deployed in the cloud. Based on the dialogue scenario text, multiple candidate scenario types are generated for the user to choose from.
[0134] Users can select specific candidate scenarios by clicking function buttons on the screen or through voice interaction. The AI chip will select the required prompts according to the user's requirements and send the voice text content, scenario text, personalized requirements, and other information to the big language model for calculation. After the big language model generates dialogue voice, it will be sent to the user's terminal interactive end for voice playback. In this way, users can conduct foreign language dialogues and exchanges with the system based on the interactive end, further improving the user's foreign language learning experience.
[0135] Example 3
[0136] This third embodiment provides a user terminal, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a user terminal provided in one embodiment of this application. It includes: a processor 01 (e.g., an AI chip), a memory 02, and a computer program stored in the memory 02 and executable on the processor (the computer program in this embodiment is at least based on the LangChain application development framework). When the processor executes the computer program, it implements the steps in the method of embodiment two of this application.
[0137] In addition, the user terminal also includes a camera 03 and a microphone 04; the camera 03 can be used to capture test questions related to foreign language learning; the microphone 04 can be used to acquire the voice content emitted by the user, and the microphone can also be used to play the dialogue voice generated by the language big model of the cloud server.
[0138] The user terminal in this embodiment also includes a sound module and a network communication module. The sound module corresponds to the microphone 04 to obtain the voice information perceived by the microphone 04.
[0139] Those skilled in the art will understand that Figure 5 This is merely an example of a user terminal in this application and does not constitute a limitation on the structure of the user terminal. It may include more or fewer components than shown in the figure, or combine certain components, or use different components.
[0140] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0141] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0142] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0144] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A system for assisting foreign language learning based on a large model, characterized in that, The system includes a user terminal and a cloud server deployed with a large language model, wherein: The user terminal is used to acquire the user's pending information, parse the pending information, and obtain target text information; The user terminal is also used to respond to user operation information regarding the target text information and send the operation information and the target text information to the cloud server; The cloud server is used to take the operation information and the target text information as input information to the language model, so that the language model can generate target information based on the input information, wherein the target information is used to assist users in learning foreign languages; Wherein, the information to be processed is a test question image related to foreign language learning; the step of obtaining the user's information to be processed includes: playing a first voice prompt and displaying the interface of the corresponding first-level function, obtaining the first function command input by the user based on the interface of the first-level function; displaying and playing a second voice prompt, collecting the test question image, and determining the question image selected by the user in the test question image; The step of parsing the information to be processed to obtain target text information includes: performing image parsing on the test question image and the question image to obtain test question text information and question text information; The response to user operation information regarding the target text information includes: responding to a second function command from the user regarding the question text information and / or the test question text information; The step of sending the operation information and the target text information to the cloud server, so that the cloud server uses the operation information and the target text information as input information for a language big model, and the language big model generates target information based on the input information, includes: determining target prompt words based on the LangChain application development framework according to the first function instruction and the second function instruction, and sending the target prompt words to the cloud server; sending the test question text information and the question text information as target text information to the cloud server, so that the cloud server uses the target prompt words and the target text information as input information for a language big model, and the language big model generates target information to assist users in learning foreign languages based on the input information; The first function instruction includes the question explanation function, the memorization function, and the foreign language dialogue function; the second function instruction includes at least the question organization function, the question explanation function, and the new question generation function.
2. The system as described in claim 1, characterized in that, The user terminal is equipped with the LangChain application development framework. The user terminal is also used to interact with the language big model of the cloud server using the LangChain application development framework, so that the cloud server uses the operation information and the target text information as input information for the language big model.
3. The system as described in claim 1, characterized in that, The user terminal is also equipped with a preset application programming interface, and the cloud server has a preset recognition model. The user terminal is also used to call the preset application programming interface to read the information to be processed, and to call the preset recognition model of the cloud server to parse the information to be processed to obtain target text information; Wherein, when the preset application programming interface is an OCR application programming interface, the preset recognition model is an OCR recognition model; when the preset application programming interface is a voice application programming interface, the preset recognition model is a voice recognition model.
4. A method for assisted foreign language learning based on a large model, characterized in that, The method is applied to a user terminal, and the method includes: Obtain the user's pending information, parse the pending information to obtain the target text information; Responding to user actions regarding the target text information; The operation information and the target text information are sent to the cloud server, so that the cloud server uses the operation information and the target text information as input information for the language big model, and the language big model generates target information based on the input information. The language big model is deployed on the cloud server, and the target information is used to assist users in learning foreign languages. The information to be processed is a test question image related to foreign language learning; The step of obtaining the user's pending information includes: Play the first voice prompt and display the corresponding first-level function interface, and obtain the first function command input by the user based on the first-level function interface; Display and play the second voice prompt, capture the test question image, and determine the question image selected by the user from the test question image; Accordingly, the step of parsing the information to be processed to obtain the target text information includes: Image analysis is performed on the test question image and the question image to obtain test question text information and question text information; The step of responding to user operation information regarding the target text information includes: Responding to a second function command from the user regarding the question text information and / or the test question text information; The step of sending the operation information and the target text information to the cloud server, so that the cloud server uses the operation information and the target text information as input information for the language big model, and the language big model generates target information based on the input information, includes: Based on the LangChain application development framework, the target prompt word is determined according to the first function instruction and the second function instruction, and the target prompt word is sent to the cloud server; The test question text information and the question text information are sent to the cloud server as the target text information, so that the cloud server uses the target prompt words and the target text information as input information for the language big model, and the language big model generates target information to assist users in learning foreign languages based on the input information; The first function instruction includes the question explanation function, the memorization function, and the foreign language dialogue function; the second function instruction includes at least the question organization function, the question explanation function, and the new question generation function.
5. The method as described in claim 4, characterized in that, The method further includes: Based on the target text information, determine the user's learning record and incorrect answers, and in conjunction with the second function instruction, generate at least one of the following tasks: multiple choice task, reading comprehension task, and fill-in-the-blank task.
6. The method as described in claim 5, characterized in that, include: A first LangChain sequence chain, a second LangChain sequence chain, and a third LangChain sequence chain are generated in advance based on the LangChain application development framework; wherein, the first LangChain sequence chain corresponds to the question organization function, the second LangChain sequence chain corresponds to the question explanation function, and the third LangChain sequence chain corresponds to the new question generation function. Accordingly, the method further includes: If the second function instruction represents the question organization function, then the first LangChain sequence chain is invoked to interact with the language big model of the cloud server; If the second function instruction represents the question explanation function, then the second LangChain sequence chain is invoked to interact with the language big model of the cloud server; If the second function instruction represents a new question function, then the third LangChain sequence chain is invoked to interact with the language big model of the cloud server.
7. The method as described in claim 4, characterized in that, The method further includes: Acquire the voice content sent by the user and convert the voice content into voice-text content; When the first function instruction represents the memorization function, the voice text content is compared with the target text information, and the comparison result is displayed after the user completes the memorization. When the first function instruction represents the foreign language dialogue function, the speech text content and the first function instruction are sent to the cloud server, so that the cloud server uses the speech text content and the first function instruction as input information for the language big model, and the language big model generates dialogue speech based on the input information.
8. A user terminal, characterized in that, The user terminal includes at least a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the method as described in any one of claims 4 to 7.
9. The user terminal as described in claim 8, characterized in that, The user terminal includes an interactive terminal and a side terminal; The interactive terminal includes at least a display screen, a camera, and a microphone; the camera is used to capture test question images related to foreign language learning; the display screen is used to display the test question images, a first voice prompt, a second voice prompt, and generated target information; the microphone is used to acquire the voice content emitted by the user, and the microphone is also used to play the dialogue voice generated by the language big model on the cloud server; The processor and the memory are deployed at the edge, and the computer program is implemented based at least on the LangChain application development framework.