Voice interaction method, device and equipment of intelligent pen and intelligent pen interaction system

Abstract

The invention relates to a voice interaction method, device and equipment of a smart pen and a smart pen interaction system, and is applied to the technical field of smart terminals. Based on a pre-trained classification model, determining a target semantic classification result corresponding to the voice information, the target semantic classification result including at least one of software and hardware scheduling, teaching resource search and knowledge questions and answers; executing a processing flow corresponding to the target semantic classification result based on the voice information to obtain a processing result; and outputting the processing result through an interaction device of the intelligent pen. The problem that in the prior art, teaching equipment is single in function is solved.

Description

Technical Field

[0001] The present application relates to the field of smart terminal technology, and in particular to a smart pen voice interaction method, apparatus, device, and smart pen interaction system. Background Art

[0002] With the rapid development of educational informatization, intelligent teaching devices are increasingly used in classrooms, and teachers are increasingly demanding ease of use and diverse functionality. Traditional teaching devices are cumbersome to operate and struggle to meet teachers' needs for efficient operation during teaching. Voice interaction technology offers a new approach to addressing this issue.

[0003] Related art teaching devices only offer simple functions, such as turning interactive devices on or off, and playing or pausing content played on interactive devices. However, when teachers need to complete other teaching tasks, they still need to divert their attention to finding and operating device functions. This cumbersome and time-consuming operation affects the rhythm and continuity of teaching. Summary of the Invention

[0004] The present application provides a smart pen voice interaction method, apparatus, device and smart pen interaction system to solve the problem of single function of teaching equipment in the prior art.

[0005] According to a first aspect of an embodiment of the present application, a voice interaction method for a smart pen is provided, comprising:

[0006] Get voice information;

[0007] Determining a target semantic classification result corresponding to the speech information based on a pre-trained classification model, wherein the target semantic classification result includes at least one of software and hardware scheduling, teaching resource search, and knowledge question answering; wherein the classification model is obtained by semantic classification training based on sample sentences and target category annotations corresponding to the sample sentences, and the target category annotations are obtained through correctness verification;

[0008] Executing a processing flow corresponding to the target semantic classification result based on the voice information to obtain a processing result;

[0009] The processing result is outputted through the interactive device of the smart pen.

[0010] Optionally, when the target semantic classification result includes the knowledge question and answer, executing the processing flow corresponding to the target semantic classification result based on the voice information to obtain the processing result includes:

[0011] Determining a target question-answering model with the strongest model operation capability from a plurality of knowledge question-answering models, wherein the model operation capability is determined by at least one of model accuracy, response time, and availability;

[0012] The voice information is input into the target question-answering model, and the target question-answering model performs semantic understanding and question-answering on the voice information to obtain the processing result, which includes the answer content corresponding to the voice information.

[0013] Optionally, when the target semantic classification result includes the teaching resource search, executing the processing flow corresponding to the target semantic classification result based on the voice information to obtain the processing result includes:

[0014] Determining a teaching resource providing platform based on the voice information;

[0015] Determining the search content indicated by the voice information, wherein the search content includes search key information and a search scope;

[0016] A search instruction is generated based on the search content and sent to the teaching resource search platform, so that the teaching resource providing platform searches within the search range according to the search key information to obtain search results.

[0017] Optionally, when the target semantic classification result includes the software and hardware scheduling, executing the processing flow corresponding to the target semantic classification result based on the voice information to obtain the processing result includes:

[0018] Extracting key scheduling information from the voice information, wherein the key scheduling information includes an operation type and an operation target;

[0019] A control instruction for the interactive device of the smart pen is generated based on the scheduling key information.

[0020] Optionally, outputting the processing result through the interactive device of the smart pen includes:

[0021] In a case where the target semantic classification result includes the knowledge question and answer, displaying the processing result in a floating window on the interactive device of the smart pen;

[0022] In a case where the target semantic classification result includes the teaching resource search, the processing result is displayed on the interactive device of the smart pen.

[0023] Optionally, the training process of the classification model includes:

[0024] Acquire an initial training sample set, where each initial training sample in the initial training sample set includes a sample sentence and an initial category label corresponding to the sample sentence;

[0025] Training multiple original models based on the initial training sample set to obtain multiple initial classification models;

[0026] For each initial training sample in the initial training sample set, the following update process is performed respectively: the sample sentences of the initial training sample are classified respectively by the multiple initial classification models to obtain multiple classification results; if more than half of the classification results are the initial category labels corresponding to the initial training sample, the initial classification label corresponding to the initial training sample is determined to be the target category label; otherwise, the correct target category label of the initial training sample is obtained; the initial category label of the initial training sample is updated to the target category label to obtain an updated training sample set;

[0027] Based on the updated training sample set, any one of the original models or the initial classification model is trained to obtain the classification model.

[0028] Optionally, obtaining an initial training sample set includes:

[0029] Obtaining an original training sample set, wherein each original training sample in the original training sample set includes a real sentence and an original category label of each real sentence;

[0030] When the number of the original training samples is less than a preset value, the original training samples are input into the large language model to obtain generated training samples, wherein the generated training samples include generated sentences and a category label for each generated sentence;

[0031] Determine a set of the original training samples and the generated training samples as the initial training set.

[0032] Optionally, the large language model is a large language model guided by a prompt word project, and the prompt word project is used to instruct the large language model to generate data of the same style according to the style of the original training sample.

[0033] According to a second aspect of an embodiment of the present application, a voice interaction device for a smart pen is provided, comprising:

[0034] An acquisition unit, configured to acquire voice information;

[0035] a classification unit, configured to determine a target semantic classification result corresponding to the speech information based on a pre-trained classification model, wherein the target semantic classification result includes at least one of software and hardware scheduling, teaching resource search, and knowledge question answering;

[0036] A processing unit, configured to execute a processing flow corresponding to the target semantic classification result based on the speech information to obtain a processing result;

[0037] The interactive unit is configured to output the processing result via the interactive device of the smart pen.

[0038] According to a third aspect of an embodiment of the present application, there is provided a smart pen interaction device, comprising a memory and a processor;

[0039] The memory is connected to the processor and is used to store programs;

[0040] The processor is configured to implement the voice interaction method of the smart pen as described in the first aspect by running the program in the memory.

[0041] According to a fourth aspect of an embodiment of the present application, there is provided a smart pen interaction system, comprising a smart pen and an interaction device of the smart pen;

[0042] The smart pen is used to collect voice information and send the collected voice to the interactive device of the smart pen;

[0043] The smart pen interaction device is configured to execute the smart pen voice interaction method as described in the first aspect.

[0044] According to a fifth aspect of the embodiments of the present application, there is provided an electronic device, including a memory and a processor;

[0045] The memory is connected to the processor and is used to store programs;

[0046] The processor is configured to implement the voice interaction method of the smart pen as described in the first aspect by running the program in the memory.

[0047] According to a sixth aspect of the embodiments of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the voice interaction method of the smart pen as described in the first aspect is implemented.

[0048] According to a seventh aspect of the embodiments of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the voice interaction method for the smart pen as described in the first aspect.

[0049] The technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application obtains voice information; determines the target semantic classification result corresponding to the voice information based on a pre-trained classification model, wherein the target semantic classification result is one of software and hardware scheduling, teaching resource search, and knowledge question and answer; executes the processing flow corresponding to the target semantic classification result based on the voice information to obtain a processing result; and feeds the processing result back to the interactive device of the smart pen. In this way, the language interaction process of the smart pen can realize multiple functional requirements including software and hardware scheduling, teaching resource search, and knowledge question and answer. By recognizing the voice information, the functional requirement indicated by the voice information can be determined from the multiple functional requirements, and the subsequent processing process can be completed, thereby reducing the tedious operation process in the teaching process, saving teaching time, and making the teaching process smoother and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0051] Figure 1 A flowchart of a voice interaction method for a smart pen is provided for one embodiment of the present application;

[0052] Figure 2 A structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] Exemplary Implementation Environment

[0055] A voice interaction system for a smart pen according to an embodiment of the present application includes a smart pen and an interaction device for the smart pen;

[0056] The smart pen is used to collect voice information and send the collected voice to the interactive device of the smart pen;

[0057] The smart pen interaction device is configured to at least perform the following steps of the smart pen voice interaction method provided in the embodiment of the present application:

[0058] Acquire speech information; determine a target semantic classification result corresponding to the speech information based on a pre-trained classification model, wherein the target semantic classification result includes at least one of software and hardware scheduling, teaching resource search, and knowledge question and answer; execute a processing flow corresponding to the target semantic classification result based on the speech information to obtain a processing result; and output the processing result through the interactive device of the smart pen. The classification model is obtained by semantic classification training based on sample sentences and the target category labels corresponding to the sample sentences, and the target category labels are obtained through correctness verification.

[0059] In some embodiments, a smart pen app can be installed on an interactive device, and the smart pen voice interaction method provided in the embodiments of this application can be executed through the smart pen app. The smart pen collects voice information and sends it to the smart pen app, which converts the voice information into text. The smart pen app can classify the voice information through the cloud (server side) and also perform intent type recognition on the text through the cloud, where the key corpus is stored. Based on the intent recognition results, the smart pen app interacts with the corresponding teaching equipment, resource library, and large language model to implement the corresponding operation.

[0060] The smart pen app can be installed on an interactive device (for example, a smart blackboard), and the interactive device can also be installed with other applications to enable PPT presentations, lesson plan annotations, video playback, etc.

[0061] Among them, the interactive equipment can be teaching equipment, such as smart blackboards and computers, or non-teaching equipment, such as fans, air conditioners, speakers, lights, etc.

[0062] The interactive device may output the processing result in, but is not limited to, image display, text display, voice broadcast, video playback, etc.

[0063] The above-mentioned smart pen interaction device executes the specific processing process of the smart pen voice interaction method provided in the embodiment of the present application. Please also refer to the processing process introduced in each embodiment of the subsequent smart pen voice interaction method.

[0064] For example, when explaining cell structure, a teacher can use voice commands to search for a 3D image of a cell. The smart pen captures the teacher's voice and sends it to the smart pen app. The smart pen app then calls a cloud server to execute the smart pen voice interaction method provided by the embodiments of this application, obtaining a 3D image of the cell. The 3D image is then displayed on the interactive device's screen, allowing students to clearly see the morphology and positional relationships of various components, such as the cell membrane, cytoplasm, and nucleus. Voice commands can also be used to display an animation demonstrating the cell division process, making abstract knowledge more intuitive and helpful for students to understand. In English grammar instruction, teachers can use voice commands to obtain various grammatical rules, and the interactive device will display the rules as a text list, such as tense formation, usage, and example sentences, and the rules for changing the singular and plural forms of nouns. In language courses such as English and Chinese, voice announcements can be used to demonstrate the pronunciation of words and texts. For example, when learning a word, the interactive device can use voice information to announce the correct pronunciation of the word, allowing students to hear standard pronunciation and correct their own pronunciation errors. When reading aloud the text, students can also use the voice playback function to feel the changes in voice, intonation, and speaking speed, and improve their sense of language.

[0065] Exemplary Methods

[0066] See also Figure 1 In an exemplary embodiment, a voice interaction method for a smart pen is provided, comprising:

[0067] Step 101: Acquire voice information.

[0068] In some embodiments, the language information may be speech collected during the teaching phase, for example, the teacher's speech during teaching or the student's speech during teaching interaction. The language information may be collected by the smart pen's microphone, or by microphones pre-installed throughout the classroom or a microphone worn by the teacher. Alternatively, the language information may be collected by microphones pre-installed throughout the classroom or a microphone worn by the teacher and played through a speaker before being collected by the smart pen's microphone. This is not specifically limited in the present embodiment.

[0069] Step 102: Based on a pre-trained classification model, determine a target semantic classification result corresponding to the voice information, wherein the target semantic classification result includes at least one of software and hardware scheduling, teaching resource search, and knowledge question and answer.

[0070] In some embodiments, a pre-trained classification model is capable of semantically classifying speech information. After acquiring speech information, it is input into the classification model, causing the classification model to output a target semantic classification result for the speech information. The classification model is trained for semantic classification based on sample sentences and the target category annotations corresponding to the sample sentences, and the target category annotations are verified for correctness.

[0071] The classification model can be, but is not limited to, a BERT model or a BERT-base model. The BERT model is an industry-proven sequence classification paradigm that combines classification performance and inference speed. The classification model can also be trained using the commonly used industry-standard fine-tuning method based on a pre-trained BERT model. For example, for Chinese sequence classification tasks, to ensure optimal fine-tuning performance on limited downstream task data, a BERT-base model pre-trained on a large Chinese corpus is selected.

[0072] The classification model is trained through semantic classification based on sample sentences and their corresponding target category annotations, which are verified for correctness. Because the target category annotations have been verified to be correct, using the correct target category annotations for model training can result in more accurate classification results from the resulting classification model.

[0073] There are two key factors that influence the effectiveness of classification model training data. First, the amount of training data—each category should have a sufficient amount of data, while also ensuring a balanced amount of data across categories. Second, the quality of the training data annotations must be as high as possible, meaning the category annotations must be as accurate as possible. Otherwise, they can mislead the model's parameter learning and even lead to non-convergence in model training. To ensure that the training data better meets training requirements, during model training, the reliability of the training data can be improved by increasing the training data and verifying the correctness of the category annotations.

[0074] Among them, when faced with insufficient training data, the following methods can be used:

[0075] Obtaining an original training sample set, wherein each original training sample in the original training sample set includes a real sentence and an original category label of each real sentence;

[0076] When the number of the original training samples is less than a preset value, the original training samples are input into the large language model to obtain generated training samples, wherein the generated training samples include generated sentences and a category label for each generated sentence;

[0077] Determine a set of the original training samples and the generated training samples as the initial training set.

[0078] The original training samples can be real sentences that actually occurred and were collected during the teaching process, and the original category labels for each real sentence can be manually annotated. Since the number of real sentences is limited, the generative capabilities of the large language model can be used to generate similar sentences.

[0079] The large language model is guided by a prompting project, and the prompting project is used to instruct the large language model to generate data of the same style according to the style of the original training sample.

[0080] A prompt is an instruction, question, or statement designed to guide or instruct a language model to produce a specific text response. Prompts are the starting point for user interaction with a language model. They clearly express the user's intent and encourage the model to respond in a valuable and relevant manner. Carefully crafted prompts can guide large language models to more accurately grasp user intent and produce more precise and useful responses. When interacting with large models, users may encounter inappropriate prompts due to lack of experience or differences between models. Therefore, prompt engineering is necessary to enable users to guide the model's output, producing highly relevant, accurate, and high-quality text.

[0081] In this way, the large language model guided by the Prompt project generates sentences of the same type and style as real sentences, obtains generated sentences, and annotates the generated sentences, thereby expanding the number of training samples. The annotation of the generated sentences can be automatically annotated by the large language model after the generated sentences are generated.

[0082] Based on the few-shot generation capability of the large language model, a batch of representative "seed data" is set for different categories of data according to the sentence expression and the knowledge points contained. Through the prompt project, the large language model is guided to generate more instruction data of the same type according to the examples of the "seed data".

[0083] Among them, few-shot is a technique applied in prompt design, which aims to guide the pre-trained language model to better context learning and task execution by providing a small number of examples.

[0084] When faced with insufficient training data (for example, the command "Search for Li Bai's 'Jiang Jin Jiu' courseware" should belong to the "teaching resource search" category, but is labeled as "command scheduling"), when such mislabeled sample data reaches a certain level, it will inevitably affect model learning. However, due to the large amount of training data, manually correcting such mislabeled data is very costly. Therefore, the following methods can be used:

[0085] Acquire an initial training sample set, where each initial training sample in the initial training sample set includes a sample sentence and an initial category label corresponding to the sample sentence;

[0086] Training multiple original models based on the initial training sample set to obtain multiple initial classification models;

[0087] For each initial training sample in the initial training sample set, the following update process is performed respectively: the sample sentences of the initial training sample are classified respectively by the multiple initial classification models to obtain multiple classification results; if more than half of the classification results are the initial category labels corresponding to the initial training sample, the initial classification label corresponding to the initial training sample is determined to be the target category label; otherwise, the correct target category label of the initial training sample is obtained; the initial category label of the initial training sample is updated to the target category label to obtain an updated training sample set;

[0088] Based on the updated training sample set, any one of the original models or the initial classification model is trained to obtain the classification model.

[0089] In some embodiments, the number of initial training samples in the initial training sample set can meet the requirements of model training. To determine whether the initial category labeling of the sample sentences in each initial training sample is correct, a "majority voting" method can be used to automatically discover and correct such labeling issues, thereby improving the overall quality of the training data.

[0090] After obtaining the initial training sample set, multiple initial classification models are randomly trained using different random seeds. For each piece of data in the initial training sample set, multiple initial classification models are used to perform inference. If more than half of the initial classification models fail to give a consistent prediction, the initial training sample is considered to be likely mislabeled or difficult to learn. These mislabeled and difficult-to-learn initial training samples are screened out and then manually relabeled to obtain their correct target category labels. The correct target category labels are then updated to the initial training sample set to obtain an updated training sample set.

[0091] The number of initial classification models may be, but is not limited to, three or more, for example, 5 or 7. To improve the effectiveness of the voting results, the number may be set to an odd number.

[0092] After obtaining an updated training sample set in which each sample sentence is correctly labeled, the corrected updated training sample set is used to train any of the original models or the initial classification model to obtain a classification model.

[0093] It is understood that determining the target semantic classification result corresponding to the language information can also be achieved through keyword matching. For example, a corpus of different classification categories can be pre-built. After obtaining the voice information, the voice information is converted into text information. Keywords in the text information are extracted and compared with the vocabulary in the corpus. If the above keywords exist in the corpus, the classification category corresponding to the corpus is determined to be the target semantic classification result of the voice information.

[0094] For example, keywords are set for software and hardware scheduling, teaching resource search, and knowledge question and answer. Software and hardware scheduling keywords include words such as "open", "close", "turn up", "turn down", "play", and "pause"; resource search keywords include words such as "find", "search", "check", "change to", and "find again"; knowledge question and answer keywords include words such as "what is", "why", "how", "how", "reason", "which", "where", "when", "how much", "which", "is there", "can", and "will"

[0095] When the voice message obtained is "open whiteboard", by identifying the keyword "open", it is judged as the intention of software and hardware function scheduling; when the voice message is "search for trigonometric function explanation videos", based on the keywords "search" and resource type, it is determined to be the intention of teaching resource search; when the voice message is "what is the Pythagorean theorem", based on the keyword "what is", it is classified as the intention of expanding knowledge question and answer.

[0096] Furthermore, the conversation context information can also be recorded, including intent type, discipline, knowledge point, resource type, etc. For example, if the user says "find a video on trigonometric functions", the system records the intent type as resource search, the knowledge point as trigonometric functions, and the resource type as video. When the next sentence is highly relevant to the current context, it is considered a continuation of the intent; if it is irrelevant, it is considered a new intent. If the user then says "preferably animation", the system will judge it as a continuation of the resource search intent and further refine the search criteria based on the previously recorded context information; if the user says "turn up the volume", it will be judged as a new software and hardware scheduling intent.

[0097] It is understood that the speech information can also be directly input into the large language model for semantic recognition, or the speech information can be first transcribed to obtain the transcribed text of the speech information, and then the transcribed text can be input into the large language model for semantic recognition. The semantic recognition here can be achieved by the large language model built into the terminal, or it can be uploaded to the cloud by the terminal and implemented by the large language model configured in the cloud.

[0098] Step 103: Execute the processing flow corresponding to the target semantic classification result based on the voice information to obtain a processing result.

[0099] Step 104: Output the processing result through the interactive device of the smart pen.

[0100] In some embodiments, there are multiple possible target semantic classification results. By configuring corresponding processing flows for different classification categories, the corresponding processing flow can be executed after determining the category to which the target semantic classification result belongs, thereby completing the interaction process. The interactive device may output the processing result by displaying the processing result or responding to a corresponding instruction and controlling a corresponding device or program to execute the instruction.

[0101] In an optional embodiment, when the target semantic classification result includes the knowledge question and answer, executing the processing flow corresponding to the target semantic classification result based on the voice information to obtain the processing result includes:

[0102] Determining a target question-answering model with the strongest model operation capability from a plurality of knowledge question-answering models, wherein the model operation capability is determined by at least one of model accuracy, response time, and availability;

[0103] Inputting the voice information into the target question-answering model, performing semantic understanding and question-answering on the voice information through the target question-answering model, and obtaining the processing result, wherein the processing result includes the answer content corresponding to the voice information;

[0104] Outputting the processing result through the interactive device of the smart pen includes:

[0105] The answer content is displayed in a floating window on the interactive device of the smart pen.

[0106] In some embodiments, multiple knowledge question answering models, such as different large language models, can be obtained from different platforms, manufacturers, and the like, and can be used to implement knowledge question answering. The operational capabilities of different knowledge question answering models may vary, and a knowledge question answering model can be selected for response based on at least one of the accuracy, response time, and availability of the model's output.

[0107] When the voice information indicates different types of questions, the target question-answering model can be selected based on the question type and the model's operational capabilities. For example, for questions that can directly output fixed answers (such as "What does 1+1 equal?"), a knowledge question-answering model with the shortest response time can be used; for questions with multiple answers (such as "What are your thoughts on Article A?"), a knowledge question-answering model with high availability can be used.

[0108] It is understandable that the voice information can also be input into multiple knowledge question and answer models separately, and the voice information can be semantically understood and answered by multiple knowledge question and answer models respectively to obtain multiple processing results, and then the multiple processing results can be integrated through the large language model to obtain the final processing result.

[0109] It is understood that if the target question-answering model is unavailable, it can automatically switch to a backup model (the backup model refers to another knowledge question-answering model other than the target question-answering model or a designated one from multiple knowledge question-answering models) and record model failure information. This failure information is fed back to technical personnel so that they can be informed in a timely manner and troubleshoot the problem.

[0110] Furthermore, after semantic understanding and question-answer responses are performed on the voice information based on the target question-answer model to obtain the processing results, since the processing results include the answer content to the voice information, the answer content can be displayed on the interactive device so that the user can know the answer to the voice information through the displayed answer content.

[0111] Among them, an answer display area can be set on the interactive device, and the answer content can be displayed in the answer display area. The answer display area can be displayed on the interactive device in the form of a floating window, and the floating window will not affect the current display content on the screen, regardless of whether the current display page on the screen is the main page of the application.

[0112] The answer content can be data in various forms such as text, pictures, and tables.

[0113] After receiving the processing results, the interactive device can also conduct a security review of the answers contained in the results to determine whether the answers contain sensitive, illegal, false, vulgar, or other negative information. If so, the answers are filtered to ensure the security of the displayed content. It can also display a watermark and a prompt stating "The relevant content is generated by the AI large language model" according to security requirements.

[0114] After the interactive device displays the processing results, the user can like or dislike the actual processing results after viewing the displayed processing results. The smart pen APP collects the feedback operations of likes or dislikes and uses them to optimize the model and content recommendation.

[0115] On the side of the content module generated by the large language model, the top three contents of Internet searches are displayed synchronously. Internet resources display website titles, content summaries or thumbnails, filter sensitive information and advertising content, and support clicking "View More" to call the browser for detailed search.

[0116] In an optional embodiment, when the target semantic classification result includes the teaching resource search, executing the processing flow corresponding to the target semantic classification result based on the voice information to obtain the processing result includes:

[0117] Determining a teaching resource providing platform based on the voice information;

[0118] Determining the search content indicated by the voice information, wherein the search content includes search key information and a search scope;

[0119] Generate a search instruction based on the search content and send it to the teaching resource search platform, so that the teaching resource providing platform searches within the search scope according to the search key information to obtain search results;

[0120] Outputting the processing result through the interactive device of the smart pen includes:

[0121] The search results are displayed on the interactive device of the smart pen.

[0122] In some embodiments, the teaching resource providing platform may be one or more of a system teaching resource library, a personal cloud disk, and an Internet resource library.

[0123] The voice information may include search actions and search content, and the search content includes search key information and search scope; wherein the search key information includes resource source, knowledge point content and resource type, and the search scope includes subject section.

[0124] By analyzing the voice message, it determines whether the resource source is specified. If not, the teaching resource provider can be selected based on the priority of the resource providers. If the resource source is specified, the teaching resource provider is directly selected according to the indication. The smart pen app returns the search intent and key information, and the teaching resource provider performs a fuzzy search from the corresponding location.

[0125] Among them, the priorities can be set in the order of system teaching resource library, personal cloud disk, and Internet resource library.

[0126] For example, the voice message is "Search for videos explaining trigonometric functions", which does not indicate the source of the resource. Therefore, the default search is in the system teaching resource library; the voice message is "Find the courseware of "The Reconciliation of the General and the Prime Minister" in my cloud disk", then the search is from the personal cloud disk.

[0127] It is understandable that when the search action is not clearly stated in the voice information, such as directly saying "video of trigonometric functions", the smart pen can automatically complete the search action based on the context.

[0128] If the search scope is indicated in the voice message, such as "high school mathematics," the search will prioritize the subject area. If no indication is given, the subject area in the user's account information will be used for auxiliary sorting. Before teaching, users can log in to the smart pen app using a pre-registered account and password and complete their personal information on the smart pen app, such as the subject area they are teaching. The smart pen app configures a personal cloud drive for each account to store personal teaching resources.

[0129] When searching, you can extract knowledge points from voice messages as keywords, supporting fuzzy matching to improve search accuracy. For example, searching for "Li Bai's Poetry Appreciation" will find relevant resources even if the keywords don't match exactly. If the command specifies a resource type, such as "video" or "courseware," the search will prioritize that type. If not, resources will be provided based on the priority of video, animation, image, audio, text, test questions, test papers, courseware, and lesson plans.

[0130] The teaching resource providing platform comprehensively considers parameters such as knowledge point relevance, resource quality (special supply > high-quality > demonstration > ordinary), time factors (within 3 months > within 6 months > within 1 year) and popularity (based on indicators such as citation volume, preview volume, and likes) to search for relevant resources. When the search results in a large amount of resource data, a preset number of resources (for example, more than 10) can be selected for recommendation to ensure the diversity and relevance of resources.

[0131] If no matching resources are found in the system's teaching resource library or personal cloud disk resources, you can search the internet resource library to display the top three resources. Internet resources include website titles, content summaries, or thumbnails, filtering out sensitive information and advertising content. Clicking "View More" allows you to call a browser for detailed search.

[0132] Since teachers themselves also need to teach in the form of oral narration, collecting classroom voice information is imperceptible to teachers and students, and classroom voice information can reflect the most authentic teaching progress in real time. The teaching resources matched in this way are more timely and accurate than those switched manually, which can effectively improve the teaching experience.

[0133] After obtaining the search results obtained from the teaching resource providing platform, the interactive device displays the search results on the interactive device for the user to view.

[0134] In an optional embodiment, when the target semantic classification result includes the software and hardware scheduling, executing the processing flow corresponding to the target semantic classification result based on the voice information to obtain the processing result includes:

[0135] Extracting key scheduling information from the voice information, wherein the key scheduling information includes an operation type and an operation target;

[0136] A control instruction for the interactive device of the smart pen is generated based on the scheduling key information.

[0137] In some embodiments, the extraction of key scheduling information can be obtained by keyword matching, etc. The smart pen transmits control instructions to the interactive device through the communication interface with the interactive device, and the operating system of the interactive device adjusts corresponding device parameters according to the control instructions.

[0138] For example, the voice message is “adjust the screen brightness to 80”, and the scheduling key information is the screen brightness and 80; the language message is “help me adjust the volume to 50”, and the scheduling key information is the volume and 50.

[0139] To launch and close applications, commands like "Open whiteboard" or "Close e-textbook" send the smart pen to the device system, enabling quick opening and closing of applications. For function calls like "Record," "Play," "Pause," and "Turn page," the smart pen interacts with the relevant application, triggering the corresponding function within the application. For example, while playing a video in a courseware, if the teacher says "Pause," the smart pen transmits the pause command to the courseware playback application, which then executes the pause operation.

[0140] It is understandable that when the target semantic classification results include more than two of software and hardware scheduling, teaching resource search, and knowledge question and answer, the above-mentioned related processing flows can be executed synchronously or asynchronously.

[0141] For example, the voice information is "What is trigonometric function? Explain it with a video." The target semantic classification results output by the classification model include teaching resource search and knowledge questions and answers. The definition of trigonometric function is output through the target question and answer model, and the definition of trigonometric function is displayed through a floating window; and through the above-mentioned search processing flow, related videos on the definition of trigonometric function are searched in the personal cloud disk, and the candidate videos are displayed on the interactive device for user selection.

[0142] The voice interaction method for the smart pen of this application accurately identifies the intentions of software and hardware function scheduling, teaching resource search, and knowledge question and answer through a classification model. It realizes efficient interaction between the smart pen and interactive devices, resource libraries, and large language models, ensuring that voice commands can be quickly and accurately converted into actual operations and services. It optimizes the teaching resource search strategy, comprehensively considers multiple factors to recommend resources, and provides Internet search as a backup to improve the accuracy and efficiency of resource acquisition. It also establishes an accurate voice command intention classification system through the construction of a keyword library, an intent classification algorithm, and a context management mechanism, combining keyword matching, context memory, and intent continuity to determine the classification category.

[0143] In this way, using the smart pen's voice command function, teachers can quickly schedule software and hardware functions and search for teaching resources without tedious manual operations, significantly saving time, making the teaching process smoother and more efficient, and improving operational efficiency. It can accurately recognize and understand the intent of complex voice commands and support a variety of command types, including software and hardware control, teaching resource searches, and expanded knowledge Q&A. It can also accurately process commands in multiple rounds of conversation, providing a more intelligent and convenient interactive experience and enhancing voice interaction capabilities. The smart pen can accurately classify voice commands and perform corresponding functions based on different intentions. Furthermore, it can remember contextual information during multiple rounds of conversation and support the refinement, correction, and deletion of commands, ensuring smooth and accurate interaction and better meeting the complex needs of the teaching process.

[0144] Exemplary devices

[0145] Accordingly, an embodiment of the present application further provides a voice interaction device for a smart pen, comprising:

[0146] An acquisition unit, configured to acquire voice information;

[0147] a classification unit, configured to determine a target semantic classification result corresponding to the speech information based on a pre-trained classification model, wherein the target semantic classification result includes at least one of software and hardware scheduling, teaching resource search, and knowledge question answering;

[0148] A processing unit, configured to execute a processing flow corresponding to the target semantic classification result based on the speech information to obtain a processing result;

[0149] The interactive unit is configured to output the processing result via the interactive device of the smart pen.

[0150] Based on any of the above embodiments, the apparatus further includes a training unit, configured to:

[0151] Acquire an initial training sample set, where each initial training sample in the initial training sample set includes a sample sentence and an initial category label corresponding to the sample sentence;

[0152] Training multiple original models based on the initial training sample set to obtain multiple initial classification models;

[0153] For each initial training sample in the initial training sample set, the following update process is performed respectively: the sample sentences of the initial training sample are classified respectively by the multiple initial classification models to obtain multiple classification results; if more than half of the classification results are the initial category labels corresponding to the initial training sample, the initial classification label corresponding to the initial training sample is determined to be the target category label; otherwise, the correct target category label of the initial training sample is obtained; the initial category label of the initial training sample is updated to the target category label to obtain an updated training sample set;

[0154] Based on the updated training sample set, any one of the original models or the initial classification model is trained to obtain the classification model.

[0155] Based on any of the above embodiments, the training unit is further configured to:

[0156] Obtaining an original training sample set, wherein each original training sample in the original training sample set includes a real sentence and an original category label of each real sentence;

[0157] When the number of the original training samples is less than a preset value, the original training samples are input into the large language model to obtain generated training samples, wherein the generated training samples include generated sentences and a category label for each generated sentence;

[0158] Determine a set of the original training samples and the generated training samples as the initial training set.

[0159] Based on any of the above embodiments, the processing unit is further configured to:

[0160] In a case where the target semantic classification result includes the knowledge question and answer, determining a target question and answer model with the strongest model operation capability from multiple knowledge question and answer models, where the model operation capability is determined by at least one of the accuracy, response time, and availability of the model; inputting the voice information into the target question and answer model, and performing semantic understanding and question and answer responses on the voice information by the target question and answer model to obtain the processing result, where the processing result includes the answer content corresponding to the voice information;

[0161] In a case where the target semantic classification result includes the teaching resource search, determining a teaching resource providing platform based on the voice information; determining search content indicated by the voice information, the search content including search key information and a search scope; generating a search instruction based on the search content, and sending the search instruction to the teaching resource search platform, so that the teaching resource providing platform searches within the search scope according to the search key information to obtain search results;

[0162] When the target semantic classification result includes the software and hardware scheduling, extract scheduling key information from the voice information, the scheduling key information including the operation type and the operation target; and generate control instructions for the interactive device of the smart pen based on the scheduling key information.

[0163] The smart pen voice interaction device provided in this embodiment is based on the same concept as the smart pen voice interaction method provided in the aforementioned embodiments of this application. It can execute the method provided in any of the aforementioned embodiments of this application and possesses the corresponding functional modules and beneficial effects. For technical details not fully described in this embodiment, please refer to the specific processing content of the smart pen voice interaction method provided in the aforementioned embodiments of this application and will not be further elaborated here.

[0164] The functions implemented by each unit in the above-mentioned voice interaction device of the smart pen can be implemented by the same or different processors, which is not limited in the embodiment of the present application.

[0165] It should be understood that each unit in the above device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device. The processor can be a general-purpose processor, such as a CPU or a microprocessor, and the memory can be a memory within the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits. The functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units can be realized by designing the logical relationships between the components within the circuit. For another example, the hardware circuit can be implemented by a PLD. For example, an FPGA can include a large number of logic gate circuits. The connection relationships between the logic gate circuits are configured through a configuration file to realize the functions of some or all of the above units. All units of the above device can be implemented entirely in the form of a processor calling software, or entirely in the form of hardware circuits, or partially in the form of a processor calling software, with the remaining parts implemented in the form of hardware circuits.

[0166] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and execute instructions, such as a CPU, a microprocessor, a GPU, or a DSP. In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0167] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0168] In addition, the various units in the above apparatus may be fully or partially integrated together, or may be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the apparatus. The at least one processor may be of different types, such as a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0169] Exemplary electronic devices

[0170] Another embodiment of the present application also provides an interactive device for a smart pen, see Figure 2 As shown, the device includes:

[0171] Memory 200 and processor 210;

[0172] The memory 200 is connected to the processor 210 and is used to store programs;

[0173] The processor 210 is configured to implement the smart pen voice interaction method disclosed in any of the above embodiments by running the program stored in the memory 200 .

[0174] Specifically, the interactive device of the smart pen may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0175] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are interconnected via a bus.

[0176] A bus may include a pathway that transfers information between components of a computer system.

[0177] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, or the like, or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. Alternatively, it can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components.

[0178] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0179] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include program code, which includes computer operating instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), other types of dynamic storage devices that can store information and instructions, disk storage, flash memory, etc.

[0180] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0181] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speakers, etc.

[0182] The communication interface 220 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0183] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement each step of any smart pen voice interaction method provided in the above embodiments of the present application.

[0184] Exemplary computer program products and storage media

[0185] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the processor executes the steps of the voice interaction method of the smart pen according to various embodiments of the present application described in any of the above embodiments of this specification.

[0186] The computer program product may be written in any combination of one or more programming languages to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0187] In addition, an embodiment of the present application may also be a storage medium having a computer program stored thereon. The computer program is used by a processor to execute the steps of the voice interaction method of the smart pen according to various embodiments of the present application described in any of the above embodiments of this specification. Specifically, the following steps may be implemented:

[0188] Get voice information;

[0189] Determining a target semantic classification result corresponding to the speech information based on a pre-trained classification model, wherein the target semantic classification result includes at least one of software and hardware scheduling, teaching resource search, and knowledge question answering; wherein the classification model is obtained by semantic classification training based on sample sentences and target category annotations corresponding to the sample sentences, and the target category annotations are obtained through correctness verification;

[0190] Executing a processing flow corresponding to the target semantic classification result based on the voice information to obtain a processing result;

[0191] The processing result is outputted through the interactive device of the smart pen.

[0192] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0193] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0194] The steps in the methods of each embodiment of the present application can be adjusted in sequence, merged, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0195] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be merged, divided, and deleted according to actual needs.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or submodules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0197] The modules or submodules described as separate components may or may not be physically separate, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules may be selected to achieve the purpose of this embodiment according to actual needs.

[0198] In addition, each functional module or submodule in each embodiment of the present application may be integrated into a processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into a single module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or software functional modules or submodules.

[0199] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0200] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software units executed by a processor, or a combination of the two. The software units may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0201] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0202] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A voice interaction method for a smart pen, characterized in that: include: Get voice information; Determining a target semantic classification result corresponding to the voice information based on a pre-trained classification model, wherein the target semantic classification result includes at least one of software and hardware scheduling, teaching resource search, and knowledge question answering; Executing a processing flow corresponding to the target semantic classification result based on the voice information to obtain a processing result; The processing result is outputted through the interactive device of the smart pen.

2. The method according to claim 1, characterized in that In a case where the target semantic classification result includes the knowledge question and answer, executing the processing flow corresponding to the target semantic classification result based on the voice information to obtain the processing result includes: Determining a target question-answering model with the strongest model operation capability from a plurality of knowledge question-answering models, wherein the model operation capability is determined by at least one of model accuracy, response time, and availability; The voice information is input into the target question-answering model, and the target question-answering model performs semantic understanding and question-answering on the voice information to obtain the processing result, which includes the answer content corresponding to the voice information.

3. The method according to claim 1, characterized in that In the case where the target semantic classification result includes the teaching resource search, executing the processing flow corresponding to the target semantic classification result based on the voice information to obtain the processing result includes: Determining a teaching resource providing platform based on the voice information; Determining the search content indicated by the voice information, wherein the search content includes search key information and a search scope; A search instruction is generated based on the search content and sent to the teaching resource search platform, so that the teaching resource providing platform searches within the search range according to the search key information to obtain search results.

4. The method according to claim 1, wherein In a case where the target semantic classification result includes the software and hardware scheduling, executing the processing flow corresponding to the target semantic classification result based on the voice information to obtain the processing result includes: Extracting key scheduling information from the voice information, wherein the key scheduling information includes an operation type and an operation target; A control instruction for the interactive device of the smart pen is generated based on the scheduling key information.

5. The method according to claim 1, wherein Outputting the processing result through the interactive device of the smart pen includes: In a case where the target semantic classification result includes the knowledge question and answer, displaying the processing result in a floating window on the interactive device of the smart pen; In a case where the target semantic classification result includes the teaching resource search, the processing result is displayed on the interactive device of the smart pen.

6. The method according to claim 1, characterized in that The training process of the classification model includes: Acquire an initial training sample set, where each initial training sample in the initial training sample set includes a sample sentence and an initial category label corresponding to the sample sentence; Training multiple original models based on the initial training sample set to obtain multiple initial classification models; For each initial training sample in the initial training sample set, the following update process is performed respectively: the sample sentences of the initial training sample are classified respectively by the multiple initial classification models to obtain multiple classification results; if more than half of the classification results are the initial category labels corresponding to the initial training sample, the initial classification label corresponding to the initial training sample is determined to be the target category label; otherwise, the correct target category label of the initial training sample is obtained; the initial category label of the initial training sample is updated to the target category label to obtain an updated training sample set; Based on the updated training sample set, any one of the original models or the initial classification model is trained to obtain the classification model.

7. The method according to claim 6, characterized in that The obtaining of the initial training sample set comprises: Obtaining an original training sample set, wherein each original training sample in the original training sample set includes a real sentence and an original category label of each real sentence; When the number of the original training samples is less than a preset value, the original training samples are input into the large language model to obtain generated training samples, wherein the generated training samples include generated sentences and a category label for each generated sentence; Determine a set of the original training samples and the generated training samples as the initial training set.

8. The method according to claim 7, characterized in that The large language model is a large language model guided by a prompt word project, and the prompt word project is used to instruct the large language model to generate data of the same style according to the style of the original training sample.

9. A voice interaction device for a smart pen, characterized in that: include: An acquisition unit, configured to acquire voice information; a classification unit, configured to determine a target semantic classification result corresponding to the speech information based on a pre-trained classification model, wherein the target semantic classification result includes at least one of software and hardware scheduling, teaching resource search, and knowledge question answering; A processing unit, configured to execute a processing flow corresponding to the target semantic classification result based on the speech information to obtain a processing result; The interactive unit is configured to output the processing result via the interactive device of the smart pen.

10. An interactive device for a smart pen, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the voice interaction method for the smart pen according to any one of claims 1 to 8 by running the program in the memory.

11. A smart pen interaction system, characterized in that: Including smart pen, and interactive device of smart pen; The smart pen is used to collect voice information and send the collected voice to the interactive device of the smart pen; The smart pen interaction device is configured to execute the smart pen voice interaction method according to any one of claims 1 to 8.

Images