A large language model-based configuration-free voice instruction dynamic mapping method and intelligent glasses

Abstract

The application relates to the field of man-machine interaction of intelligent wearable devices, in particular to a configuration-free voice instruction dynamic mapping method based on a large language model and intelligent glasses; the method comprises the following steps: S100, function registration, receiving registration information, and generating a function registration list; S200, voice collection and recognition, collecting voice signals sent by a user and converting the voice signals into voice texts; S300, semantic understanding and dynamic mapping, a large language model performs the following operations based on semantic matching: a model outputs a structured instruction, a natural language reply or a parameter inquiry request according to a matching result; and S400, instruction distribution and execution, a system distributes and executes or broadcasts feedback after analyzing the output of the large language model; the application realizes a configuration-free voice control ecology, supports natural spoken language expression of a user, and has intelligent conflict resolution capability.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology for smart wearable devices, specifically to a configuration-free dynamic mapping method for voice commands based on a large language model and smart glasses. Background Technology

[0002] With the advancement of speech recognition and natural language processing technologies, voice control has become one of the mainstream interaction methods for smart devices. Currently, the voice control functions of most smart devices (such as smart speakers, smart home devices, smartphones, and smart glasses) heavily rely on the skill configuration platforms provided by third-party voice service providers (such as iFlytek, Alibaba, Google Assistant, Baidu DuerOS, etc.). The general workflow is as follows: (1) The device picks up the audio and transmits the audio stream to the cloud of the voice service provider; (2) The cloud-based automatic speech recognition (ASR) module converts the audio into text; (3) The cloud-based natural language understanding (NLU) module performs intent recognition and slot filling based on the pre-configured "skills"; (4) Instruction mapping is performed according to the intent and slots to generate structured instructions that the device can execute; (5) The instructions are returned to the device and executed by the corresponding functional modules.

[0003] In this mode, adding any new voice control function requires the device manufacturer or application developer to manually create a "skill" in the voice service provider's backend. This involves tedious steps: defining the intent (such as "open a book"), configuring a large amount of possible user speech for the intent (such as "I want to read that book", "open Romance of the Three Kingdoms", "continue reading", etc.), specifying the entity / slot (such as "book title"), and setting the cloud response or command sent to the device to trigger the intent. In addition, the voice feedback content such as prompts and confirmations on the device also needs to be pre-written in text form in the backend and then synthesized and broadcast by the cloud TTS module.

[0004] This configuration is acceptable for home appliances with relatively simple functions (such as smart fans). However, for smart glasses with highly rich functions and a continuously expanding application ecosystem, the drawbacks are extremely significant: First, the maintenance cost is extremely high, and plug-and-play functionality is not possible. Smart glasses have numerous third-party applications installed (such as reading, navigation, instant messaging, etc.), each of which may offer a variety of voice-controlled functions. If it is required to continuously add skills, corpora, and slots to the voice service provider's backend for every function of every application, the workload of operation and maintenance will be enormous, and any update of any application or installation of a new application will cause backend configuration to lag, resulting in extremely poor system scalability.

[0005] Second, user expression is limited, resulting in a rigid experience. Users must accurately pronounce commands that closely match the preset vocabulary for the skill; if the expression is colloquial, vague, or slightly off in terms of function, the system cannot recognize it. Users must memorize preset, fixed commands and cannot perform natural, flexible voice operations.

[0006] Third, the feedback is rigid. The voice feedback content on the device must be pre-programmed in the skill configuration, making it impossible to generate timely, rich, and personalized responses based on the specific context of the user's command, which makes the interaction seem mechanical and unintelligent.

[0007] Fourth, conflict resolution is difficult. If multiple applications register similar voice control functions (such as the "navigation" function in map and food delivery applications), traditional skill configuration systems lack intelligent and dynamic conflict resolution mechanisms.

[0008] Therefore, how to provide a voice control method that can completely solve the above-mentioned problems of configuration-free operation, dynamic mapping, natural interaction, and intelligent feedback has become a technical problem that needs to be solved. Summary of the Invention

[0009] To overcome the aforementioned technical problems in the prior art, this invention provides a configuration-free dynamic mapping method for voice commands based on a large language model and smart glasses. This solves the technical pain points in the voice control methods of existing smart glasses and other devices, which require pre-configuring skills, corpora, and slots in the cloud for each specific control function, resulting in poor system scalability, high development and maintenance costs, limited user command expression, rigid interactive feedback, and difficulty in resolving functional conflicts.

[0010] To achieve the above objectives, this invention provides a configuration-free dynamic mapping method for voice commands based on a large language model, applied to smart glasses, comprising: Step S100: Function registration, receiving function registration information from a service or application, generating and dynamically maintaining a function registration list; the function registration information includes a unique identifier of the service or application, at least one function name that can be invoked by voice, a text description of the function, and parameter names and parameter types required to execute the function; Step S200: Voice acquisition and recognition, after the voice assistant is activated, acquiring the voice signal emitted by the user, and calling an automatic speech recognition model to convert the voice signal into speech text; Step S300: Semantic understanding and dynamic mapping, combining the speech text and the function registration list at the current moment into prompt words, and sending them into the large language model; the large language model performs the following operations based on semantic matching: if a match is successful... If the target function is matched and the required parameters are complete, a structured instruction containing the target function name and parameter key-value pairs is output; if the target function is matched but at least one required parameter is missing, a parameter follow-up request is output; if no registered function can be matched, a natural language response text is output; Step S400: Instruction distribution and execution, parsing the output of the large language model: if it is the structured instruction, the corresponding function registration entry is found according to the function name, and the call instruction carrying parameters is sent to the corresponding target application or system service for execution; if it is the natural language response text, the text-to-speech model is used to synthesize speech and broadcast it to the user; if it is the parameter follow-up request, the follow-up prompt is broadcast through the text-to-speech model, secondary sound pickup is initiated to obtain the parameter value supplemented by the user, the supplemented complete parameter is merged with the original speech text and sent back to the large language model to generate a complete structured instruction and distribute it for execution.

[0011] Preferably, the function registration information in step S100 is implemented in the following way: the third-party application makes a remote procedure call through the interface bound to the voice assistant service, the definition of which includes methods for registering, deregistering, and updating functions; the function registration information is transmitted using a structured request body, the request body including the application's unique identifier, the application's display name, and a function list; each function object in the function list includes a function name, function description, parameter definition, function category, and conflict resolution priority prompt; after receiving a valid request, the voice assistant service atomically updates its internal function registration list and returns a response including an operation success status, a message, and an error code.

[0012] Preferably, in step S300, the prompt word further includes the following: a format restriction instruction to constrain the large language model to output only pure JSON format and prohibit the addition of any explanatory text; constraint rules to guide parameter extraction, including entity values ​​should be extracted from the user's original input text and should not be fabricated, and normalization of enumeration type parameters to standard values ​​in a colloquial style; and information of the current system front-end application, including the front-end application package name and application name, to guide matching priority in case of functional conflicts.

[0013] Preferably, in step S300, the structured data in a preset format output by the large language model includes one of the following four types: Type 1, Function Instruction Mapping: includes a type identifier, target application identifier, function name, extracted parameter key-value pairs, and matching confidence; Type 2, Natural Language Response: includes a type identifier and response text for broadcast; Type 3, Parameter Follow-up: includes a type identifier, target application identifier, function name, and a list of missing parameters; each item in the list of missing parameters includes a parameter name, parameter description, and follow-up prompt text for broadcast; Type 4, Function Conflict: includes a type identifier, a unique conflict identifier, a list of candidate functions, prompt text for broadcast, timeout time, and selection mode; each item in the list of candidate functions includes a candidate index, application identifier, application name, function name, function description, and extracted parameters.

[0014] Preferably, when the large language model outputs the parameter follow-up structure of type three, step S400 further includes a secondary sound pickup process: synthesizing the follow-up prompt text into speech through the text-to-speech module and broadcasting it to the user, and starting a secondary sound pickup window for the missing parameters; collecting the user's supplementary parameter value speech and converting it into supplementary text; merging the supplementary text with the original speech text into a context, and sending it back to the large language model for parameter extraction, generating complete structured data and distributing it for execution.

[0015] Preferably, it also includes intelligent function conflict adjudication: when multiple registered functions constitute a potential match based on the user's voice text, a conflict adjudication mechanism is triggered; the conflict adjudication mechanism includes a foreground priority strategy: the application package name currently running in the foreground of the smart glasses is injected as context into the prompt word, and the large language model integrates the user input, function list, and foreground information to output the final structured data containing the adjudication result in one go; the conflict adjudication mechanism also includes a user selection strategy: when the conflicting applications are not in the foreground or are unrelated to the foreground applications, the large language model outputs the function conflict structure of type four; the conflict prompt and options are broadcast through the text-to-speech module, secondary sound pickup is initiated to receive the user's selected voice, the user's selection is parsed, the corresponding function information is extracted from the candidate function list, the function instructions are assembled, and they are distributed for execution.

[0016] Preferably, the intelligent adjudication of functional conflicts further includes a context management mechanism for secondary interactions: the voice assistant service maintains a session context manager in memory, assigns a unique session identifier to each dialogue cycle from wake-up to instruction completion, and maintains a state object; when the large language model outputs a functional conflict structure of type four, the current session state is set to a waiting state for user selection, conflict data is stored, a text-to-speech prompt text is played, and a timeout pickup window is started; the user's selected voice is sent to the selection intent parser, the selected candidate index is parsed, the corresponding function information is retrieved according to the index, and the function is distributed for execution; after successful parsing or timeout, the session state is reset to an idle state.

[0017] Preferably, the method further includes step S500, dynamic function management: receiving a function deletion or modification request and updating the function registration list; the function deletion or modification request is triggered by the user through a specific voice command, or by the application calling the deregistration or update interface when uninstalling or updating.

[0018] Accordingly, the present invention also provides smart glasses, employing the method provided by the present invention, comprising: a sound pickup module for acquiring voice signals emitted by a user; a speaker module for broadcasting voice feedback; a communication module for data interaction with a cloud service; a processor and a memory; the processor running a voice assistant service, the voice assistant service integrating or capable of calling an automatic speech recognition model, a text-to-speech model, and a large language model; the memory storing executable instructions, the processor executing the instructions to implement the method steps of any one of claims 1-8.

[0019] Preferably, the automatic speech recognition model, the text-to-speech model, and the large language model are all deployed in the cloud and invoked through the communication module; or they are deployed in the memory and executed locally by the processor.

[0020] The present invention has at least the following technical effects through the technical solution provided by the present invention: This invention completely eliminates the reliance on manual configuration of background skills, corpora, and slots. Developers and operating system services only need to register "what I can do" through a lightweight API, without any natural language processing knowledge, to incorporate their functions into the voice assistant's controllable scope. This greatly lowers the barrier to entry for smart glasses applications into the ecosystem, enabling the basic functions of any newly installed app to be instantly controlled by voice. When users install new applications, the system adds new services, or existing functions change, the corresponding voice control capabilities are immediately understood by the LLM through dynamic updates to the function registration list, without waiting for any background updates to take effect, achieving instant expansion of system capabilities. Users do not need to memorize or recite any fixed commands; they can express their needs in the most natural and conversational way. The powerful LLM can accurately map various expressions such as "Let's look at the book I didn't finish reading last time" or "I want to continue reading The Three-Body Problem" to the "open the book" function and extract the parameter "The Three-Body Problem".

[0021] This invention leverages the generative capabilities of LLM (Limited Language Management) to prevent a cold error message when a user's intent cannot be mapped to a control command. Instead, it generates witty and helpful chat responses or inquiries, which are then read back via TTS (Text-to-Speech). Furthermore, the Agent can dynamically generate feedback before and after command execution, rather than playing a pre-set confirmation tone. This invention also combines a "foreground application priority" and "voice option guidance" adjudication strategy, elegantly and accurately solving the command distribution problem when multiple applications offer similar functions without requiring manual user intervention. In particular, it ensures the continuity of operations on the user's current visual focus (foreground application). This invention also requires the LLM to output strictly formatted structured data (ISON), rather than secondary parsing of natural language. This completely encapsulates the uncertainty of speech understanding within the Agent, ultimately distributing deterministic and parsable machine instructions to the application, ensuring absolute reliability and security for cross-application calls. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the general process of the configuration-free dynamic mapping method for voice commands based on a large language model provided by the present invention; Figure 2 This is a schematic diagram illustrating the function registration and maintenance of the dynamic function list in an embodiment of the present invention; Figure 3 This is a complete flowchart of a certain embodiment of the present invention, including parameter inquiry, conflict resolution, and secondary interaction; Figure 4This is a schematic diagram of the architecture for the interaction between smart glasses and cloud services in an embodiment of the present invention. Detailed Implementation

[0023] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0024] In this invention, the terms "system" and "network" are used interchangeably. "Multiple" refers to two or more; therefore, in this invention, "multiple" can also be understood as "at least two." "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, it should be understood that in the description of this invention, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0025] Please see Figure 1 This invention provides a configuration-free dynamic mapping method for voice commands based on a large language model, applied to smart glasses. The smart glasses have a built-in or accessible voice assistant, an automatic speech recognition (ASR) model, a text-to-speech (TTS) model, and a customized large language model (LLM). The method includes: Step S100: Function registration. Receive function registration information from a service or application, generate and dynamically maintain a function registration list; the function registration information includes the unique identifier of the service or application, at least one function name that can be invoked by voice, a text description of the function, and the parameter names and parameter types required to execute the function. Step S200: Voice acquisition and recognition. After the voice assistant is woken up, the voice signal emitted by the user is acquired, and the automatic speech recognition model is called to convert the voice signal into speech text. Step S300: Semantic understanding and dynamic mapping, combining the spoken text and the current function registration list into prompt words, and feeding them into the large language model; the large language model performs the following operations based on semantic matching: If the target function is successfully matched and the required parameters are complete, a structured instruction containing the target function name and parameter key-value pairs will be output. If a target function is matched but at least one required parameter is missing, a parameter follow-up request is output; if no registered function is matched, a natural language response text is output. Step S400: Instruction distribution and execution, parsing the output of the large language model: if it is the structured instruction, then find the corresponding function registration entry according to the function name, and send the call instruction carrying parameters to the corresponding target application or system service for execution; If the response is in natural language, then the text-to-speech model is invoked to synthesize speech and broadcast it to the user. If the parameter is requested for further inquiry, the text-to-speech model will play the prompt, initiate secondary sound pickup to obtain the parameter value supplemented by the user, merge the supplemented complete parameter with the original speech text, and then send it back into the large language model to generate a complete structured instruction and distribute it for execution.

[0026] In this invention, step S100 involves receiving function registration information from an operating system-level service or a third-party application, generating and dynamically maintaining a function registration list. This function registration information does not involve configuring any corpus or slots with the voice service provider; it only includes a unique identifier for the service or application, at least one function name that can be invoked by voice, a text description of the function, and the parameter names and types required to execute the function. Specifically, the unique identifier can be the application's package name (parameter is the book name, type is a string), "page turning" (parameter is the page turning direction, type is a string, value can be forward or backward), "add bookmark," etc. This registration can be completed by the application initiating a simple application programming interface (API) call to the voice assistant service residing in the glasses system during installation or first startup, and the voice assistant adds the entry to the function registration list. Furthermore, the API needs to be defined in a way that is easy for developers to access; it is recommended to use the Android system's ContentProvider or Bound Service approach, balancing standardization and lightweight cross-process communication. Preferably, it is implemented using a Bound Service combined with the Android interface definition language. Specifically, third-party applications perform remote procedure calls by binding to the Voice Assistant Agent service's interface. This interface definition includes methods for registration, deregistration, and update functions. The Voice Assistant service registers as a globally bindable Android service upon system startup and returns an object implementing the registration interface in its callback method. Function registration information is transmitted using a structured request body; this request body contains the following fields: Application Unique Identifier (string, required, used to uniquely identify the application initiating registration); Application Display Name (string, required, used to display the application's name to the user); and Function List (array, required, containing one or more function objects registered by the application).

[0027] Furthermore, each function object contains the following fields: Function Name (string, required, unique within the same application); Function Description (string, required, describing the function's purpose in natural language for semantic matching by the large language model); Parameter Definition (object, required, where the key is the parameter name and the value is an object containing the parameter type and description); Function Category (string, optional, used to categorize functions, such as content consumption, navigation, system control, etc.); Conflict Resolution Priority Hint (string, optional, value is either normal or high priority, used to assist in resolving function conflicts). Upon receiving a valid registration request, the voice assistant service atomically updates its internal function registration list and returns a response. The response includes the following fields: a boolean value indicating whether the operation was successful, a prompt message, an error code, and the number of functions registered this time. The error code system is defined as follows: Successful operation is the first code; invalid request body format or missing required fields is the second code; an empty or invalid application identifier is the third code; duplicate function names within the same application are the fourth code; incorrect parameter definition format is the fifth code; the function registration list is full exceeding the system limit is the sixth code; and the application is not authorized to call the voice assistant service is the seventh code. The above interface also applies to the deregistration function (deregistering all registered functions of an application based on the application identifier) ​​and the update function (updating the registered function list of an application, actually performing an atomic operation of first deleting and then adding).

[0028] In this invention, in step S200, after the voice assistant is woken up, the voice signal emitted by the user is collected, and the local or cloud-based ASR model is called to convert the voice signal into speech text in real time.

[0029] In this invention, in step S300, the voice text and the current function registration list (or a set of all function names and descriptions therein) are combined into a prompt word, which is then fed into a large language model. The large language model is trained to perform the following operations through careful prompt word engineering and targeted instruction follow-up fine-tuning: To understand the intent expressed by the user's voice and text; Perform semantic-level matching of this intent with the names and text descriptions of each feature in the feature registration list; If a unique and specific target function is successfully matched, the parameter values ​​required to execute the function are extracted, and structured data that strictly follows a preset format is output; the structured data includes at least the name of the matched target function and parameter key-value pairs extracted from the speech text; If no registered function can be matched (i.e., the user input is non-control intent such as casual conversation or knowledge quiz), then a natural language response text will be generated directly. If a target function can be matched, but at least one parameter necessary to execute the function is missing or its value cannot be determined in the speech text, the large language model should not forcibly generate an incomplete structured instruction, nor should it directly determine it as a casual question and answer. Instead, it should generate a parameter follow-up request structure, and the system should ask the user for the missing parameter through voice broadcast, guide the user to supplement the parameter, supplement the parameter after secondary sound pickup, and finally synthesize a complete instruction.

[0030] A complete template for prompt words consists of the following parts: Role setting section: The large language model is instructed to act as a command mapping agent for smart glasses devices. Its task is to accurately map the user's voice input text to a specific function in the list of available functions and extract the required parameters; if it cannot be mapped to any function, it needs to generate natural and friendly response text.

[0031] Output format restrictions: The large language model is strictly required to output only a single, strictly defined JSON object when it can be mapped to a usable function, without any explanatory text, greetings, or characters other than punctuation before or after the JSON. When it cannot be mapped to any function, it should output only a JSON object with a type field of "chat" and a response field of natural language response text. When a function can be matched but the required parameters are missing, it should output only a JSON object with a type field of "parameter follow-up question." Outputting any content other than the three JSON formats mentioned above is strictly prohibited.

[0032] Available Function List: A list of all currently schedulable functions is dynamically injected at runtime. Each function includes its application identifier, function name, function description, and required parameter definitions.

[0033] Parameter extraction constraints: The large language model must carefully analyze user input and extract entity values ​​that match the functional parameter definitions. Entity values ​​should be original text fragments from user input or short values ​​that have been normalized. It is not allowed to fabricate or speculate on content not mentioned by the user. For string type parameters, extract the text value directly. For numeric type parameters, extract the number and convert it to numeric type. For Boolean type parameters, determine whether it is true or false based on semantics. If the parameter definition contains enumeration constraints, the user's colloquial expression must be normalized into standard enumeration values.

[0034] Parameter extraction constraint rules: 1. Carefully analyze user input and extract entity values ​​that match the functional parameter definitions. Entity values ​​should be original fragments from user input or short, normalized values; do not fabricate or speculate on content not mentioned by the user.

[0035] 2. For string type parameters, extract the text value directly; for number type parameters, extract the number and convert it to a numeric type; for boolean type parameters, determine whether it is true or false based on semantics.

[0036] 3. If the parameter definition contains enumeration constraints (such as direction can be next or previous), you must normalize the user’s colloquial expressions (such as “turn to the next page” or “turn to the previous page”) into enumeration values ​​(“next” or “previous”).

[0037] 4. When a functional conflict occurs (multiple functions constitute a potential match), if there is a conflict resolution message, it should be taken into consideration; otherwise, all potential matching functions should be listed in the output JSON using the format "conflict" for type.

[0038] Foreground application information section: The application package name and application name currently being used by the user in the foreground are injected at runtime. This is used to prioritize the selection of the foreground application as the matching result if a function from this foreground application exists among the candidate functions in the event of a function conflict.

[0039] User input section: Injects the text content after speech recognition of the current user at runtime.

[0040] The structured data output by the large language model in its preset format includes one of the following four types: Type 1 is a function instruction mapping: This structure contains a type identifier, a target application identifier, a function name, extracted parameter key-value pairs, and an optional matching confidence level (within the range of zero to one).

[0041] Type 2 is a natural language response: This structure contains a type identifier and the response text to be broadcast.

[0042] Type 3 is parameter follow-up: This structure includes a type identifier, a target application identifier, a function name, and a list of missing parameters; each item in the list of missing parameters includes a parameter name, a parameter description, and follow-up prompt text for broadcast.

[0043] Type 4 is a function conflict: This structure includes a type identifier, a unique conflict identifier, a list of candidate functions, a prompt text for broadcast, a timeout period, and a selection mode; each item in the candidate function list includes a candidate index, an application identifier, an application name, a function name, a function description, and extracted parameters.

[0044] For fine-tuning the large language model, a supervised fine-tuning approach is used to customize the training of the basic large language model. The training dataset is constructed as follows: The training set contains four categories of samples proportionally. The first category, approximately 50%, consists of normal function matching samples, comprised of user-generated verbal commands paired with correct Type 1 JSON output. The second category, approximately 20%, consists of non-control intent samples, comprised of user input such as casual conversation or knowledge-based questions paired with correct Type 2 JSON output. The third category, approximately 20%, consists of user input containing functional intent but lacking necessary parameters, paired with correct Type 3 JSON output. The fourth category, approximately 10%, consists of ambiguous commands that simultaneously match multiple functions, paired with correct Type 4 JSON output.

[0045] For function matching samples, a diversified rewriting strategy is adopted: for each registered function, a large language model is used to help generate no less than fifty different colloquial expressions of user instructions, covering natural language variants such as direct instructions, ambiguous instructions, and instructions with redundant information, to ensure that the model has good generalization ability for different expressions of the same function.

[0046] For samples with missing parameters, for each function containing required parameters, construct user input that expresses the intention of the action but deliberately omits all or part of the required parameters, and mark it as the correct parameter follow-up JSON output.

[0047] For negative sample training, construct some user inputs whose functional intent does not exist in the function registration list, label them as type two, i.e., natural language response, and provide reasonable rejection or guiding responses.

[0048] To strictly constrain training by output format, a penalty mechanism is introduced during training, applying higher loss weights to the generated results that output additional explanatory text before and after JSON, ensuring that the model strictly adheres to the format restriction of only outputting pure JSON during inference.

[0049] In this invention, step S400 involves parsing the data output by the large language model. If it is type one function instruction mapping structured data, the corresponding function registration entry is found based on the function name. The call instruction carrying parameters is then sent to the corresponding target application or system service according to the application identifier. The target application or service then executes the corresponding function and presents the result. This call can be based on Android's intent mechanism, callback interface, or inter-service communication.

[0050] If the output of the large language model is a type 2 natural language response text, then the text-to-speech model is called to synthesize the natural language response text into speech and broadcast it to the user.

[0051] If the large language model outputs a type 3 parameter follow-up request structure, the text-to-speech module synthesizes the follow-up prompt text or dynamically concatenated follow-up statements into speech and plays it to the user, initiating a secondary sound pickup process for the missing parameter. After the user provides the missing parameter value, the system merges that parameter value with the original speech text to form context, and then sends it back to the large language model for parameter extraction, ultimately generating complete structured data and distributing it for execution.

[0052] Furthermore, the method of the present invention also includes a functional conflict intelligent adjudication step, which is performed during the matching process of the large language model in step S300, or as a separate post-processing step. When multiple registered functions constitute a potential match based on the user's voice text, the conflict adjudication mechanism is triggered.

[0053] Specifically, the conflict resolution mechanism includes a foreground priority strategy: before invoking the large language model, the voice assistant service's command distribution module queries the application package name of the currently running foreground activity on the smart glasses via the Android system's activity manager interface or usage statistics manager interface. This information is used as a context-injected prompt. The implementation employs a two-stage inference strategy to avoid increasing the number of calls to the large language model. This operation is performed before each semantic understanding step in voice command processing, and the obtained foreground application package name is cached in the assistant service's current session context. Subsequently, when constructing the prompt, the assistant service injects the foreground application package name and corresponding application name as the "current system foreground application" field, directly informing the large language model to prioritize the foreground application's function. The large language model, after integrating user input, the function list, and foreground information, outputs the final structured data containing the resolution result in one go.

[0054] The conflict resolution mechanism also includes a user selection strategy: if none of the conflicting applications are in the foreground, or if the foreground applications are irrelevant, the large language model does not subjectively select from the output structured data, but instead generates a type four functional conflict structure. The instruction distribution module broadcasts these options to the user through the text-to-speech module and initiates a brief secondary sound pickup.

[0055] The function conflict structure includes the following fields: a unique conflict identifier, used to associate subsequent user selections; a list of candidate functions, each containing a candidate index number, application identifier, application name, function name, function description, and extracted parameters; a prompt text for announcement; a timeout period for waiting for secondary user input, after which the system will automatically cancel or default to selecting the first candidate; and a selection method, which can be either voice selection or touch selection.

[0056] The context management mechanism for secondary interaction is as follows: A session context manager is maintained in memory through the voice assistant service, assigning a unique session identifier to each dialogue cycle from wake-up to command completion, and maintaining a state object. When the large language model outputs the type four functional conflict structure, the command distribution module performs the following operations: sets the current session state to a waiting state for user selection and stores the conflict data; plays the prompt text from text to speech and starts a pickup window with a timeout duration; sends the user's selected speech to a lightweight selection intent parser to parse out the selected candidate index; retrieves the corresponding functional information from the candidate list of conflict data based on the selected index, assembles the type one functional command mapping structure, and distributes it for execution; after successful parsing or timeout, the session state is reset to an idle state.

[0057] Furthermore, the method also includes a function management step S500: receiving a function deletion or modification request and updating the function registration list. This operation can be triggered by the user through a specific voice command, or by the application during uninstallation or update by calling the application programming interface, ensuring the real-time nature and accuracy of the function registration list.

[0058] This invention also provides smart glasses employing the above method, comprising: a sound pickup module for acquiring voice signals emitted by a user; a speaker module for broadcasting voice feedback; a communication module for data interaction with a cloud service; a processor; and a memory. The processor runs a voice assistant service, which integrates or can invoke an automatic speech recognition model, a text-to-speech model, and a large language model. The memory stores executable instructions, and the processor executes these instructions to implement any of the steps described above. The automatic speech recognition model, text-to-speech model, and large language model can be deployed in the cloud and invoked through the communication module; alternatively, they can be partially or entirely deployed in local memory and executed locally by the processor.

[0059] Example 1 This embodiment details a configuration-free dynamic mapping method for voice commands based on a large language model, the process of which is as follows: Figure 1 As shown.

[0060] In the background of the smart glasses' operating system, there runs a "Voice Assistant Agent Service," which serves as the central hub for all voice interactions. This service integrates the ability to call various large models.

[0061] During the registration phase, such as Figure 2As shown, an e-book reading application with the package name com.smart.reader is installed and launched by the user. Upon first launch, the application sends a function registration request by calling the registration function method through the binding service interface provided by the system. This request contains the following information: the application identifier is the package name of the reading application, the application display name is "Smart Reading," and the function list contains two function items.

[0062] The first function is named "Open Book" and described as "Open a specified book". The parameter definition includes a parameter named "Book Name", which is of type string and described as "the name of the book".

[0063] The second function is named "Page Turning" and described as "Page Turning Forward or Backward". The parameter definition includes a parameter named "Page Turning Direction", which is a string and described as "Page Turning Direction, with a value of forward or backward".

[0064] Simultaneously, the system service also registered general functions, such as "adjust volume" (parameter is volume level, type is numerical) and "adjust brightness" (parameter is brightness level, type is numerical). The voice assistant service merges and deduplicates these registration requests, generating a persistent list of registered functions. This process does not require developers to configure corpora or slots on any cloud platform.

[0065] During operation, the user wakes up the smart glasses and says, "Hey glasses, I want to continue reading the book 'The Three-Body Problem' that I read last time." Voice Acquisition and Recognition: The system picks up the voice and calls a high-performance cloud-based automatic speech recognition model to convert the speech into text: "I want to continue reading the same book 'The Three-Body Problem' from last time." Semantic understanding and dynamic mapping: The voice assistant service queries the current foreground application (assuming it is a reading application), and then assembles the foreground information, the complete function registration list, and the text according to the prompt word template before feeding them into the large language model. After semantic matching, the large language model identifies that the user's intent is to execute the "open book" function of the reading application, and extracts the value of the parameter "book name" as "Three-Body Problem". Subsequently, it outputs a type-one function instruction mapping structure.

[0066] Command distribution and execution: The voice assistant service parses the structured data, finds the reading application based on the application identifier, and passes the parameters through the intent mechanism or callback. The application opens the "Three-Body Problem" reading interface, and the interaction is completed.

[0067] Example 2 This embodiment focuses on illustrating the intelligent adjudication mechanism for functional conflicts, and its specific process is as follows: Figure 3 As shown.

[0068] Suppose a user has both a map app and a lifestyle services app installed on their glasses, and each app has registered a function: the map app has registered a "start route navigation" function (with the destination as the parameter), and the lifestyle services app has registered a "view delivery navigation for takeout orders" function (with the order identifier as the parameter). The user wakes up the glasses and says, "Navigate to the company."

[0069] The large language model identified two potential matching functions, and the current foreground application is a map application (the user is viewing a map). Since the prompt word already contains foreground information, the large language model directly applies a foreground-first strategy, outputting a type-one function instruction mapping structure. The target application is the map application, the function is "Start Route Navigation," and the parameter is the destination value, which is "Company." The instruction is then dispatched and executed.

[0070] If the user utters the same command while the photo app is in the foreground, the large language model will output a type four function conflict structure because the foreground priority condition is not met. This structure includes a unique conflict identifier, two candidate functions (path navigation for a map app and food delivery navigation for a local services app), and a broadcast prompt text. The broadcast prompt text is: "Multiple programs have been detected that can achieve this. Say 'one' or 'first' to use map navigation, and say 'two' or 'second' to check food delivery." The voice assistant service enters a secondary voice pickup state and plays a prompt. The user says "first," and the parser parses the selected candidate index one, extracts the navigation function from the map application and the extracted parameter information, assembles the final type-one function instruction mapping structure, and executes it. If a timeout occurs or a new wake word appears, the operation is canceled. This mechanism intelligently and seamlessly resolves semantic ambiguity in multi-application environments and perfectly follows the user's current operational context.

[0071] Example 3 This example illustrates the dynamic management of functions.

[0072] To remove the voice-activated page-turning feature from a reading app, a user can say, "Assistant, please don't use voice to turn pages anymore." The large language model maps this intention to the system's built-in function management metafunction. The voice assistant service updates the function registration list, marking the reading app's page-turning entry as disabled or removed. Afterward, when a user says "turn to the next page," the large language model will not consider this function during matching, potentially indicating that it cannot be executed or instead executing another similar function. When the app is uninstalled, the system will also automatically clear all its registered function items, keeping the list clean.

[0073] Example 4 This embodiment illustrates the hardware architecture and deployment method of smart glasses, such as... Figure 4 As shown.

[0074] The smart glasses provided by this invention include the following hardware modules: The pickup module uses a microphone array to collect the user's voice signal, supports far-field pickup and noise suppression, and ensures that the user's voice can still be clearly collected in complex acoustic environments.

[0075] The speaker module is used to broadcast voice feedback and supports real-time playback of text-to-speech synthesized speech.

[0076] The communication module supports Wi-Fi, Bluetooth, and cellular networks for data interaction with cloud services.

[0077] The processor is responsible for running the voice assistant service, executing command distribution, managing the function registration list, and other core logic.

[0078] The memory stores executable instructions, and optionally stores locally deployed automatic speech recognition models, text-to-speech models, and large language models.

[0079] Regarding deployment methods, the automatic speech recognition model, text-to-speech model, and large language model in this invention can adopt one or a combination of the following two deployment methods: Cloud deployment: The model is deployed on a cloud server, and the smart glasses access services by calling the cloud application programming interface through the communication module. This method is suitable for large-scale models with high computational resource requirements, and can leverage the powerful computing capabilities of the cloud to provide high-quality speech recognition, semantic understanding, and speech synthesis capabilities.

[0080] Local deployment: The model is partially or entirely deployed in the smart glasses' memory and executed locally by the processor. This method is suitable for scenarios with high real-time requirements or unstable network environments, providing basic voice interaction capabilities even without a network connection.

[0081] Edge-cloud collaboration approach: A hybrid deployment strategy of edge-cloud collaboration is adopted, with lightweight models running locally to handle common commands, while complex semantic understanding tasks call large models in the cloud, ensuring both response speed and understanding accuracy.

[0082] Example 5 This embodiment illustrates the complete process of functional conflict structure and secondary interaction.

[0083] The functional conflict structure contains the following fields: The conflict unique identifier is a string type used to uniquely identify this conflict event, facilitating association with subsequent user selections.

[0084] The candidate feature list is an array, where each candidate contains: a candidate index number (to identify the candidate's position in the list), an application identifier, an application name, a feature name, a feature description, and parameters extracted from the user input.

[0085] The prompt text is a string type, which is used by the text-to-speech module to synthesize speech and read it to the user to guide the user in making a choice.

[0086] The timeout period is a numeric value representing the maximum time in milliseconds the system will wait for a second user input. After the timeout, the system will either automatically cancel the operation or select the first candidate by default.

[0087] The selection method is a string type, with values ​​of either voice selection or touch selection, used to indicate how the system receives the user's selection input.

[0088] The context management mechanism for secondary interactions is as follows: The voice assistant service maintains a session context manager in memory. This manager assigns a unique session identifier for each dialogue cycle from wake-up to command completion and maintains the corresponding state object.

[0089] After the large language model outputs a type 4 functional conflict structure, the instruction dispatch module executes the following process: First, set the current session state to "waiting for user selection" and store the conflict data in the session context manager.

[0090] Secondly, the text prompt is read aloud via the text-to-speech module, while a voice pickup window with a continuous timeout period is started, waiting for the user to make a selection via voice.

[0091] Then, the user's voice selection is sent to the selection intent parser, which can understand natural language expressions such as "first", "second", "one", "two" and parse out the index number of the candidate option selected by the user.

[0092] Finally, based on the parsed index number, the corresponding functional information is retrieved from the candidate list of conflicting data, assembled into a type-one functional instruction mapping structure, and distributed to the corresponding target application for execution. If parsing is successful or times out, the session state is reset to idle.

[0093] If the user then says, "Open a book," the large language model recognizes the "Open a book" function but lacks the "book name" parameter. It will output a type three parameter follow-up question structure, which includes the missing parameter name as "book name," the parameter description as "book name," and the follow-up question text as "Which book would you like to open?" The voice assistant service uses text-to-speech: "Which book would you like to open?" The user adds "The Three-Body Problem," and the system merges the two texts and re-enters them into the large language model, ultimately generating a complete type-one function instruction mapping structure and executing it.

[0094] If a user asks, "What do you think of this book?", since no registered functions can be matched, the large language model outputs a Type II natural language response structure, with the response text being an introductory evaluation of the science fiction masterpiece "The Three-Body Problem". The voice assistant service calls the text to speech synthesis and playback, and the entire interaction is smooth and intelligent, requiring no cloud-based skill configuration.

[0095] Furthermore, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the present invention.

[0096] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention.

[0097] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not describe the various possible combinations separately.

[0098] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0099] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.

Claims

1. A configuration-free dynamic mapping method for voice commands based on a large language model, applied to smart glasses, characterized in that, include: Step S100: Function registration: Receive function registration information from services or applications, generate and dynamically maintain a function registration list; The function registration information includes a unique identifier for the service or application, at least one function name that can be invoked by voice, a text description of the function, and the parameter names and parameter types required to execute the function. Step S200: Voice acquisition and recognition. After the voice assistant is woken up, the voice signal emitted by the user is acquired, and the automatic speech recognition model is called to convert the voice signal into speech text. Step S300: Semantic understanding and dynamic mapping, combining the spoken text and the current function registration list into prompt words, and feeding them into the large language model; the large language model performs the following operations based on semantic matching: If the target function is successfully matched and the required parameters are complete, a structured instruction containing the target function name and parameter key-value pairs will be output. If the target function is matched but at least one required parameter is missing, output a parameter follow-up request; If no registered feature can be matched, output a natural language response text; Step S400: Instruction dispatch and execution, parsing the output of the large language model: If it is a structured instruction, then the corresponding function registration entry is found according to the function name therein, and the call instruction carrying parameters is sent to the corresponding target application or system service for execution; If the response is in natural language, then the text-to-speech model is invoked to synthesize speech and broadcast it to the user. If the parameter is requested for further inquiry, the text-to-speech model will play the prompt, initiate secondary sound pickup to obtain the parameter value supplemented by the user, merge the supplemented complete parameter with the original speech text, and then send it back into the large language model to generate a complete structured instruction and distribute it for execution.

2. The method according to claim 1, characterized in that, The function registration information in step S100 is implemented in the following way: Third-party applications make remote procedure calls through an interface bound to a voice assistant service. The definition of the interface includes methods for registration, deregistration, and update functions. The function registration information is transmitted using a structured request body, which includes a unique application identifier, an application display name, and a function list. Each function object in the function list includes a function name, function description, parameter definition, function category, and conflict resolution priority prompt. Upon receiving a legitimate request, the voice assistant service atomically updates its internal function registration list and returns a response containing the operation success status, a message, and an error code.

3. The method according to claim 1, characterized in that, In step S300, the prompt word also includes the following: Formatting restrictions are used to constrain the large language model to output only pure JSON format and prohibit the addition of any explanatory text; The constraints used to guide parameter extraction include that entity values ​​should be extracted from the original user input and should not be fabricated, and that enumeration type parameters should be normalized to standard values ​​in a colloquial way. Information about the current system's foreground applications, including the foreground application package name and application name, is used to guide matching priorities in case of functional conflicts.

4. The method according to claim 1, characterized in that, In step S300, the structured data in the preset format output by the large language model includes one of the following four types: Type 1, Function Instruction Mapping: Includes type identifier, target application identifier, function name, extracted parameter key-value pairs, and matching confidence; Type 2, Natural Language Response: Includes a type identifier and the response text to be broadcast; Type 3, Parameter Follow-up: Includes type identifier, target application identifier, function name, and a list of missing parameters; each item in the list of missing parameters includes parameter name, parameter description, and follow-up prompt text for broadcast. Type 4, Functional Conflict: Includes type identifier, conflict unique identifier, candidate function list, prompt text for broadcast, timeout time, and selection mode; each item in the candidate function list includes candidate index, application identifier, application name, function name, function description, and extracted parameters.

5. The method according to claim 4, characterized in that, When the large language model outputs the type 3 parameter probing structure, step S400 further includes a secondary sound pickup process: The text-to-speech module synthesizes the follow-up question text into speech and plays it to the user, and activates a secondary sound pickup window for the missing parameters; Collect the user's additional parameter values ​​via voice and convert them into supplementary text; The supplementary text is merged with the original speech text to form a context, which is then fed back into the large language model for parameter extraction, generating complete structured data and distributing it for execution.

6. The method according to claim 4, characterized in that, It also includes intelligent adjudication of functional conflicts: When multiple registered functions constitute a potential match based on the user's voice text, the conflict resolution mechanism is triggered. The conflict resolution mechanism includes a foreground priority strategy: the application package name currently running in the foreground of the smart glasses is injected as context into the prompt word, and the large language model integrates user input, function list and foreground information, and outputs the final structured data containing the resolution result at one time. The conflict resolution mechanism also includes a user selection strategy: when the conflicting applications are not in the foreground or are unrelated to the foreground applications, the large language model outputs the functional conflict structure of type four; the conflict prompts and options are broadcast through the text-to-speech module, secondary sound pickup is initiated to receive the user's selected voice, the corresponding function information is extracted from the candidate function list after parsing the user's selection, the function instructions are assembled and distributed for execution.

7. The method according to claim 6, characterized in that, The intelligent adjudication of functional conflicts also includes a context management mechanism for secondary interactions: The voice assistant service maintains a session context manager in memory, which assigns a unique session identifier and maintains a state object for each dialogue cycle from wake-up to command completion. After the large language model outputs a type four functional conflict structure, the current session state is set to a state of waiting for user selection, the conflict data is stored, a text-to-speech prompt text is played, and a timeout sound pickup window is started. The user's selected voice is sent to the selection intent parser, which parses out the selected candidate index, retrieves the corresponding function information based on the index, and distributes it for execution. If the resolution is successful or times out, reset the session state to idle.

8. The method according to claim 1, characterized in that, It also includes step S500, dynamic function management: Receive requests to delete or change features, and update the feature registration list accordingly; The request to delete or change the function is triggered by the user through a specific voice command, or by the application calling the logout or update interface when it is uninstalled or updated.

9. A smart pair of glasses, employing the method as described in any one of claims 1-8, characterized in that, include: The microphone module is used to collect the voice signals emitted by the user. Speaker module, used for broadcasting voice feedback; The communication module is used for data interaction with cloud services; Processor and memory; The processor runs a voice assistant service, which integrates or can invoke automatic speech recognition models, text-to-speech models, and large language models. The memory stores executable instructions, and the processor executes the instructions to implement the method steps of any one of claims 1-8.

10. The smart glasses according to claim 9, characterized in that, The automatic speech recognition model, the text-to-speech model, and the large language model are all deployed in the cloud and invoked through the communication module; or they are deployed in the memory and executed locally by the processor.

Images