Intelligent earphone interaction method and system based on large language model intelligent assistant agent

Abstract

The application discloses an intelligent earphone interaction method and system based on a large language model intelligent assistant Agent. The method comprises the following steps: scanning and constructing a tool interface registry; deploying a large language model service of a server-local end architecture and a sub-intelligent assistant Agent specially used for processing various device operation transactions; capturing a user voice instruction and dispatching a corresponding sub-intelligent assistant Agent to generate a processing instruction; and querying the tool interface registry by the local end service and dynamically calling a related interface by using a code reflection mechanism to execute the processing instruction. The application realizes complex and accurate control of an intelligent earphone and associated devices through a natural language, and improves the intelligent level of interaction and user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent wearable devices and human-computer interaction, and particularly relates to an intelligent earphone interaction method and system based on a large language model intelligent assistant Agent. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the large language model (LLM) has become the core technology for building a new generation of intelligent assistants due to its powerful natural language understanding and generation capabilities. As an important wearable device, intelligent earphones are considered as an ideal entry for voice interaction with LLM assistants due to their portability and privacy. Users expect to use natural language instructions through intelligent earphones to conveniently control various application functions on mobile terminals (such as smartphones and tablet computers), such as sending messages, setting reminders, controlling smart homes, etc.

[0003] However, there are still several significant challenges in the current technical background to deeply integrate LLM assistants into smart earphones to achieve seamless control of mobile terminals: First, the size and response delay of existing smart assistants (Agents). To achieve universality, traditional LLM smart assistants are usually designed to be extremely large and complex, and they need to cover a large number of possible operations. When such Agents are deployed in the cloud, the response needs to go through a complete loop of "user device-cloud-user device". Network delay and cloud computing queuing inevitably lead to a decrease in response speed, making it difficult to meet the user's millisecond response expectation for immediate control (such as "turn on flashlight" and "play music"). If such a large Agent is deployed completely locally on a mobile terminal with severely limited computing resources, storage space, and battery capacity, it is also not feasible. Second, the privacy protection defects of the end-cloud collaborative architecture. Common cloud LLM service solutions require uploading the user's voice instructions and even personal data necessary for executing the instructions (such as contact lists, schedule information, and local files) to the server for processing. This approach poses a serious risk of privacy leakage. Mobile terminals contain a large amount of highly sensitive personal information, and users generally have concerns about transmitting it to third-party servers. Although existing technologies propose some local processing solutions, they are often limited to simple speech recognition (ASR), and the subsequent intent understanding and task execution still rely on the cloud, making it difficult to truly achieve "data does not leave the device" and meet the growing privacy and security needs of users. Third, the lack of flexibility and scalability of function integration. The functions of existing mobile terminal voice assistants (such as the built-in voice assistant of a smartphone) are usually predefined and hard-coded, with poor scalability. When a user wants to integrate a new application or service, they often need to wait for official support from the operating system or assistant manufacturer, and cannot add it flexibly by the developer or user. The system lacks a standardized mechanism for dynamically discovering, registering, and calling various application and service interfaces on the mobile terminal, resulting in the inability of the LLM assistant to fully utilize the device's own ecosystem.

[0004] Therefore, for the specific application scenario of smart earphones and mobile terminals, there is an urgent need for a novel technical solution that can achieve a lightweight, responsive, and extensible LLM smart assistant interaction system while ensuring low latency and high privacy security.

[0005] The information disclosed in the Background section of the present invention is only intended to deepen the understanding of the general background of the present invention, and should not be considered as acknowledging or implying in any form that this information constitutes prior art known to those skilled in the art. SUMMARY

[0006] The application provides an intelligent earphone interaction method and system based on a large language model intelligent assistant Agent, which can realize deep understanding of user voice instructions by the intelligent earphone and autonomous device operation, and provides an efficient, safe and personalized intelligent interaction experience.

[0007] In a first aspect, the application provides an intelligent earphone interaction method based on a large language model intelligent assistant Agent, which comprises the following steps:

[0008] S100, scanning a tool interface provided by a local system to obtain an interface ID and an interface name, finding an interface description corresponding to the interface name, and combining and registering the interface ID, the interface name and the interface description in a tool interface registry;

[0009] S110, deploying a large language model LLM service system based on a server-client architecture, receiving a user registration application, wherein the registration application information contains a tool interface registry; clustering tool interfaces according to functions based on the interface description in the tool interface registry, and setting a sub-intelligent assistant Agent of the large language model LLM based on the function clustering result, wherein the sub-intelligent assistant Agent is used to process device operation transactions of the corresponding functions;

[0010] S120, when the intelligent earphone is connected to the local system, starting a local end background service of the large language model LLM, collecting a voice instruction received by the user's intelligent earphone, and sending the voice instruction to the server end, calling the large language model LLM to analyze the voice instruction, determining a function sequence to be executed for the voice instruction according to the analysis result, and calling the corresponding sub-intelligent assistant Agent according to the function type in the function sequence to process;

[0011] S130, returning the generated processing instruction to the local end by the sub-intelligent assistant Agent, querying the tool interface corresponding to the processing instruction in the tool interface registry by the local end background service, and dynamically calling the tool interface through a code reflection mechanism to realize the response of the user instruction.

[0012] Preferably, the step S100 comprises: S101 determining the system version of the local operating system OS, searching the official SDK API document, obtaining the interface ID and interface name based on the official SDK API document, and searching the interface description corresponding to the interface name; S102 scanning the API provided by the local system application and the third-party application through the Content Provider interface to obtain the interface ID and the interface name, searching the interface description of the service document provided by the local system application and the third-party application based on the interface ID and the interface name, and searching the interface description in the Internet based on the interface name; S103 performing a supplementary scanning step to obtain the interface ID, the interface name, and the interface description corresponding to the interface name; and S104 combining the interface ID, the interface name, and the interface description into a triple (id, name, des) and registering in the tool interface registry.

[0013] Preferably, the interface description at least includes permission level information and interface calling mode information used by the interface, and the interface ID, the interface name, the interface description, the permission level information, and the interface calling mode information are combined into a quadruple (id, name, des, perm, schema) and registered in the tool interface registry.

[0014] Preferably, the step S110 specifically comprises: S111, using an embedding model to convert the interface descriptions of all tools in the tool interface registry into semantic vectors, and then determining a function cluster by combining tools with similar functions based on a clustering analysis model, and defining a type for each function cluster; S112, based on the type of the function cluster, writing a system prompt word for a sub-intelligent assistant Agent; and S113, maintaining a system prompt word for the sub-intelligent assistant Agent, generating a sub-intelligent assistant Agent instance based on a basic large language model LLM and the system prompt word, and using the sub-intelligent assistant Agent instance to process a device operation transaction corresponding to the function.

[0015] Preferably, the step S120 specifically comprises: S121, deeply understanding the intent; S122, generating a function sequence in combination with the intent; and S123, routing the function sequence to a corresponding sub-intelligent assistant Agent, and the sub-intelligent assistant Agent analyzes the function sequence and generates a response instruction.

[0016] In a second aspect, the embodiments of the present disclosure provide an intelligent earphone interaction system based on a large language model intelligent assistant Agent, comprising:

[0017] An interface registration module is configured to scan tool interfaces provided by a local system to obtain an interface ID and an interface name, search for an interface description corresponding to the interface name, and combine the interface ID, the interface name, and the interface description and register them in a tool interface registry.

[0018] An agent deployment module is configured to deploy a large language model (LLM) service system of a server-client architecture, receive a user registration application, and obtain tool interface registration information from the registration application. The tool interface registration information is used to cluster tool interfaces according to interface descriptions in the tool interface registration information, and set sub-intelligent assistants (Agents) of the LLM based on the clustering results. The sub-intelligent assistants are configured to process device operation transactions of corresponding functions.

[0019] A voice instruction processing module is configured to start a local background service of the LLM when an intelligent earphone is connected to a local system, collect voice instructions received by the intelligent earphone, and send the voice instructions to a server. The server is configured to analyze the voice instructions based on the LLM, determine a function sequence to be executed based on the analysis results, and call corresponding sub-intelligent assistants based on function types in the function sequence.

[0020] A processing instruction execution module is configured to return processing instructions generated by the sub-intelligent assistants to the local system, query tool interfaces corresponding to the processing instructions in a tool interface registration table, and dynamically call the tool interfaces based on a code reflection mechanism to respond to user instructions.

[0021] Preferably, the interface registration module includes:

[0022] A first scanning unit is configured to determine a system version of a local operating system (OS), find an official SDK API document, obtain an interface ID and an interface name based on the official SDK API document, and find an interface description corresponding to the interface name.

[0023] A second scanning unit is configured to scan APIs provided by local system applications and third-party applications through a Content Provider interface to obtain an interface ID and an interface name, find an interface description of a service document provided by the local system applications and the third-party applications based on the interface ID and the interface name, and find an interface description on the Internet based on the interface name.

[0024] A supplementary scanning unit is configured to perform a supplementary scanning step, obtain an interface ID and an interface name, and find an interface description corresponding to the interface name.

[0025] A registration unit is configured to combine the interface ID, the interface name, and the interface description into a triple (id, name, des) and register the triple in a tool interface registration table.

[0026] Preferably, the interface description at least includes permission level information of interface use and interface calling mode information, the permission level information of interface use and the interface calling mode information are combined with the interface ID, the interface name and the interface description to form a quadruple (id, name, des, perm, schema), and are registered in the tool interface registry.

[0027] Preferably, the Agent deployment module specifically includes:

[0028] a function clustering unit, configured to convert interface descriptions of all tools in the tool interface registry into semantic vectors using an embedding model, and then determine tools with similar functions to form function clusters based on a clustering analysis model, and define a type for each function cluster;

[0029] a prompt word management unit, configured to write system prompt words for the sub-intelligent assistant Agent based on the type of the function cluster;

[0030] an assistant instantiation unit, configured to maintain system prompt words for the sub-intelligent assistant Agent, and generate an instance of the sub-intelligent assistant Agent based on a basic large language model (LLM) and the system prompt words, for processing device operation transactions of corresponding functions.

[0031] Preferably, the voice instruction processing module specifically includes:

[0032] an understanding unit, configured to deeply understand an intent of a user voice instruction;

[0033] a function unit, configured to generate a function sequence to be executed in combination with the understood intent;

[0034] a routing unit, configured to route the generated function sequence to a corresponding sub-intelligent assistant Agent, and analyze the function sequence and generate a response instruction by the sub-intelligent assistant Agent.

[0035] The above scheme at least achieves the following technical effects: (1) By the tool interface scanning and registration mechanism and the sub-intelligent assistant allocation scheme based on functional clustering, the general large language model is constrained to multiple lightweight and specialized sub-intelligent assistant agents. Each sub-intelligent assistant agent only needs to handle transactions in a specific functional domain, and its instruction set and decision logic are greatly simplified, greatly reducing the dependence on cloud computing resources and significantly optimizing the control-response delay of the intelligent earphone, (2) The end-cloud collaborative architecture of cloud analysis and local execution ensures that private data does not leave the local and authorized execution of sensitive functions, ensuring user privacy and building a security barrier. (3) The scanning and registration mechanism ensures the scalability of the intelligent earphone control function; it breaks through the limitations of the existing mobile terminal voice assistant function solidification and poor scalability, and solves the problem that it cannot automatically identify and utilize the functions of third-party applications installed on the user device. BRIEF DESCRIPTION OF DRAWINGS

[0036] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures, and in which:

[0037] Figure 1 is a flowchart of an intelligent earphone interaction method based on a large language model intelligent assistant agent according to an embodiment of the present application.

[0038] Figure 2 is a structural block diagram of an intelligent earphone interaction system based on a large language model intelligent assistant agent according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0040] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application and above-mentioned drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the use of the terms so construed herein is merely for convenience and only to aid in understanding the application and is in no way a limitation on its broader scope. It is also to be understood that the application can be practiced in other than the described embodiments, which are presented for purposes of illustration and not of limitation.

[0041] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the present application with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0042] In a first aspect, the embodiments of the present disclosure provide an intelligent earphone interaction method and system based on a large language model intelligent assistant Agent, the method comprising the steps of:

[0043] S100, scanning a tool interface provided by a local system, obtaining an interface ID, an interface name, finding an interface description corresponding to the interface name, and combining and registering the interface ID, the interface name, and the interface description in a tool interface registry;

[0044] In the field of artificial intelligence (AI), Agent refers to a computer system or software entity that can perceive its environment, make autonomous decisions, and perform actions to achieve specific goals. Its core features are autonomy and goal orientation. When combined with a large language model (LLM), the intelligent agent uses the LLM as the core reasoning engine to guide its decision-making process, and through tool invocation, it expands the capability boundary of the LLM, enabling it to evolve from a pure text generator to an autonomous actor that can interact with the digital and physical worlds and perform complex tasks. However, when performing tasks, Agent first and must obtain the functional interfaces that the device (or environment) can provide, which is a prerequisite for the correct operation of Agent. Therefore, the core mechanism of the tool interface registry in S100 is provided, which scans the tool interfaces provided by the local system and processes them to obtain a tool interface registry for configuring Agent functions.

[0045] In a preferred embodiment, the step S100 comprises: S101 determining the system version of the local operating system OS, searching the official SDK API document, obtaining the interface ID and interface name based on the official SDK API document, and searching the interface description corresponding to the interface name; S102 scanning the API provided by the local system application and the third-party application through the Content Provider interface to obtain the interface ID and interface name, searching the interface description of the service document provided by the local system application and the third-party application based on the interface ID and interface name, and searching the interface description and interface application in the Internet based on the interface name; S103 performing a supplementary scanning step to obtain the interface ID, interface name, and interface description corresponding to the interface name; S104 combining the interface ID, interface name, and interface description into a triple (id, name, description) and registering in the tool interface registry. Further, in a preferred embodiment, the interface description at least includes the permission level used by the interface, the permission level information is combined with the interface ID, interface name, and interface description into a quadruple (id, nam, des, perm), and is registered in the tool interface registry. Further, in a preferred embodiment, the interface description at least includes the interface calling mode information, the interface calling mode information is combined with the permission level information, interface ID, interface name, and interface description into a quadruple (id, nam, des, perm, schema), and is registered in the tool interface registry.

[0046] In the above embodiments, S101 is a discovery method for the operating system native API. The API of different OS versions (such as Android 10 vs Android 14) will be added, changed or removed. By first determining the system version, and then locating the official SDK document of the corresponding version, the specific information of the system-level function can be most accurately obtained, such as the alarm, dialing, address book, GPS, Bluetooth, etc. interface, to build a tool registry of system-level functions. S102 is a discovery method for system applications or third-party applications. System applications or third-party applications can open more tool interfaces to the outside to more richly expand device functions. For example, in the Android system, the Content Provider is a standard mechanism for sharing data between applications (the similar concept in iOS is URL Schemes or App Intents). By scanning the system-registered Content Provider, it can be found which application provides which callable data or operation, so as to extend the Agent's ability from the operating system to the third-party application ecosystem, ensuring its scalability. For example, "Alipay" may provide a Content Provider for payment. In S103, the scanning tool interface technology is still being updated stably, so the interface information can also be supplemented in multiple ways, for example, through the URL Schemes or App Intents mechanism, analyzing application service call interfaces, decompiling APK, parsing application metadata components, etc. The potential scanning tool interface technology is used as a supplementary scanning step to perfect the interface data obtained. In S104, the data obtained is stored in a structured manner; in the preferred embodiment, the permission level is added, which provides a key basis for subsequent security decisions, and the Agent or system can decide whether to pop up a window to the user for permission or directly refuse to perform high-risk operations. In the preferred embodiment, the interface calling method information is added, which can be implemented in JSON format, used to describe how to fill in the parameters, how to call the interface, how to set the calling constraints, etc., which is a further supplement to the interface description, and is conducive to the correct calling of the caller Agent. For example, for the function of sending a short message, its schema value can be:

[0047] json

[0048] {

[0049] "type": "object",

[0050] "properties": {

[0051] "phone_number": {

[0052] "type": "string",

[0053] "description": "The recipient's phone number"

[0054] },

[0055] "message": {

[0056] "type": "string",

[0057] "description": "The content of the text message"

[0058] }

[0059] },

[0060] "required": ["phone_number", "message"]

[0061] }

[0062] In one embodiment, step S100 occurs in the pre-deployment phase, and the tool interface registry is dynamically updated. The method further includes a listening step for listening to application installation, uninstallation, update events and system update events. When a change is detected, an incremental scan is triggered to update only the changed part, thereby efficiently ensuring system scalability.

[0063] S110, deploying a large language model LLM service system of a server-client architecture, receiving a user registration application, the registration application information containing a tool interface registry; according to the interface description in the tool interface registry, the tool interfaces are clustered by function, and a large language model LLM is set based on the function clustering result. The sub-intelligent assistant Agent is used to process the device operation transaction of the corresponding function.

[0064] In the prior art, a large and general LLM in the cloud is usually used to complete all work, including understanding the intention, planning the steps, and calling the built-in tools to execute, etc. The general LLM is not adaptive to mobile terminal control, for example, it is insufficient in controlling the ability of local devices, and it has defects in computing efficiency and professionalism.

[0065] The application still adopts server-side-local side architecture, however, there is essential difference in the end cloud cooperation between the server side and the local side, that is, constructing efficient and special Agent for the mobile phone control of the intelligent earphone. The cloud service receives the registration application from the user mobile terminal, and the key information is the tool interface registration table, which completely describes all the operable functions of the device. The cloud uses NLP (natural language processing) and machine learning technology to perform semantic analysis on the "interface description" of all tools in the registration table, and automatically classifies the tools with the same or similar functions into a category. The tools in the same category are usually the tools provided by the mobile terminal with similar or cooperative functions, such as setting an alarm clock and finding a date, which have cooperative functions, making a phone call and hanging up a phone call, which have similar functions, WeChat payment and Alipay payment, which have similar functions. According to the results of cluster analysis, the cloud does not create a complete large Agent, but initializes multiple sub-intelligent assistant Agents. Each sub-intelligent assistant Agent is trained or configured to handle a specific field / specific category of task, becomes a "field expert", which makes the response more accurate and fast; the sub-intelligent assistant Agent can understand the tools and functions that the mobile terminal can provide and the scenarios that the mobile terminal can realize, which makes the instruction processing more specialized.

[0066] In a preferred embodiment, step S110 specifically includes: S111, using an embedding model to convert the interface descriptions of all tools in the tool interface registry into semantic vectors, and then performing a clustering analysis model to combine tools with similar functions to form functional clusters, and defining the type of each functional cluster. S112, based on the type of the functional cluster, write system prompts for the sub-intelligent assistant Agent, including: role definition, core instruction, capability range, parsing protocol, constraint condition. Modern LLMs (such as GPT-4) work based on context (Context), and system prompts (System Prompt) are the most direct and effective means to shape their behavior. Through precise prompts, it can be configured as a functional specialist focusing on a specific field. This has been verified in some applications (such as Copilot, ChatGPT's specific mode). In an embodiment, the tool interface registry clustering obtains three types of functions, namely communication tools, media tools, and payment tools. Taking the communication class as an example, the role definition can be: a communication class sub-intelligent assistant, responsible for handling all user requests related to mobile phone communication; the core instruction includes: following the template process of thinking-planning-calling, planning the tool sequence to be called, and finally outputting the structured instructions of the calling function; the capability range includes: tools that can be called: {name: call_phone;des: dial a specified parameter phone;para:num}, {name: search_contacts;des: query mobile phone contacts;para:name}, etc., which can be listed one by one according to the tool interfaces in the communication function cluster; the parsing protocol is used to force the LLM to output in a specific format (such as JSON) for program parsing; the constraint condition configures operation permissions, function constraints, and other parameters. S113, maintain a prompt template for each sub-intelligent assistant Agent, and based on the basic large language model LLM and the prompt template, constitute a running sub-intelligent assistant Agent instance for handling device operation transactions corresponding to the function.

[0067] In a preferred embodiment, a plurality of conventional types of sub-intelligent assistant Agents are pre-configured on the server side, and the sub-intelligent assistant Agent based on the function clustering result setting large language model LLM is specifically: directly calling the pre-configured sub-intelligent assistant Agent to constitute a running sub-intelligent assistant Agent instance.

[0068] S120, when the intelligent earphone is connected to the local system, the local end background service of the large language model LLM is started, the local end background service collects the voice instruction received by the user's intelligent earphone, and sends it to the server end, the server end calls the large language model LLM to analyze the function of the voice instruction, determines the function sequence to be executed according to the analysis result, and calls the corresponding sub intelligent assistant Agent according to the function type in the function sequence to process.

[0069] This step describes the complete process from the user issuing a voice instruction to the cloud LLM planning a specific operation sequence. The core is to use the powerful reasoning ability of the cloud LLM to understand the instruction and plan a function sequence executed by the sub-agent, and convert the user's intention into a specific action decision cycle.

[0070] With a specific embodiment, when the smart earphone is connected to the local system, the local end background service of the large language model LLM is started, the local end background service collects the voice instruction received by the user's smart earphone, and sends it to the server end. S121: Deeply understand the intention. In one embodiment, the user issues a voice instruction through the smart earphone: "Please ask me how long it will take to get to the hospital", the server end LLM receives the instruction "I will arrive at the hospital" after the text is converted, and the server end LLM deeply understands the intention: the core target is to calculate the estimated time (ETA) to reach the hospital; Key missing parameter 1: "Hospital" is which one? It can be inferred according to the context or preference (for example, the last one, the one designated by the user's medical insurance, the one the user often goes to. Key missing parameter 2: "I" where? The user's real-time current location needs to be obtained. Key calculation logic: ETA = Route planning engine calculates according to the starting point (user location) and ending point (hospital location), combined with real-time traffic conditions. S122: Generate function sequence combining intention. The function sequence is, for example: "get the user's accurate latitude and longitude coordinates", "infer which specific location the user refers to as 'hospital'", and "navigation function". S123: Route the function sequence to the corresponding sub-intelligent assistant Agent, and the sub-intelligent assistant Agent analyzes the function sequence and generates a response instruction. The navigation sub-Agent (in some cases, according to the different function sequences, multiple sub-intelligent assistant Agents may cooperate to handle a transaction) can handle the above function sequence, which analyzes the function sequence and generates instructions: positioning instructions, querying the nearest first-class hospital instructions, querying road state instructions, starting navigation instructions, calculation instructions, and voice playing instructions. When there are privacy and permission requirements, the above instructions can be sent to the mobile terminal for execution at the same time; when there is no privacy requirement, the above instructions can be partially executed in the cloud (such as positioning (with privacy requirements), querying (without privacy requirements), and navigation in the cloud processing, navigation (with privacy requirements), and voice (without privacy requirements) playing locally).

[0071] S130, the sub-intelligent assistant Agent returns the generated processing instructions to the local end, and the local end background service queries the tool interface corresponding to the processing instructions in the tool interface registry, and dynamically calls the tool interface through the code reflection mechanism to realize the response of the user's instruction.

[0072] This step describes the process of how the processing instructions generated by the sub-intelligent assistant Agent are finally executed on the user device, bringing the intelligent planning in the cloud to the specific actions on the device, and completing the function call locally in a safe and efficient manner. Specifically, after receiving the task, the sub-intelligent Agent generates specific and executable processing instructions, which are usually a structured command containing the tool name and specific parameters to be called. The processing instructions are issued to the local background service of the mobile phone. After receiving the instructions, the local service queries the local tool interface registry to find the real interface ID (i.e. the API exposed by the system or application) corresponding to the tool interface specified in the instructions. Through the code reflection mechanism, the system dynamically calls and executes a series of interface APIs to complete the operation the user wants, and then feeds back the result to the user after the operation is completed.

[0073] The code reflection mechanism can find and call function entities by function name. For example, the tool for sending emails is executed in the following way:

[0074] String methodName = "sent_email";

[0075] Method method = clazz.getMethod(methodName, String.class);

[0076] String result = (String) method.invoke(instance, "Hello");

[0077] / / Output: Send Hello success

[0078] In a preferred embodiment, after querying the tool information, the local service should check the 'perm' permission field. If the permission level is "normal", it will be executed directly. If the permission level is "dangerous" (for example, send_sms, read_contacts), it will query whether the operation is authorized by the user, and suspend execution or continue execution according to the query result.

[0079] In a preferred embodiment, the parameters generated by the sub-intelligent assistant (Agent) are preprocessed locally to ensure they meet the requirements for tool invocation. For example, in an email sending operation, the parameter "Xiao Wang" is a nickname and cannot be directly used as an email address. Before calling the `send_email` function, the local machine automatically calls another registered tool, `query_contact`, to resolve "Xiao Wang" into the specific email address ABC@example.com. This process is transparent to the cloud-based LLM, reducing the burden on the LLM while ensuring the security of local data.

[0080] In a preferred embodiment, after dynamically invoking the tool interface via code reflection, an intermediate result is obtained. This intermediate result is then sent as a parameter to the corresponding sub-intelligent assistant agent on the server side to execute the next round of planning and execution. In a preferred embodiment, if successful, a voice command is invoked and broadcast to the user via TTS (Text-to-Speech) to the headset, ending the process. In case of failure or exception, the local end will capture the exception and return an error message (such as "No contact named Xiao Wang found") to the cloud. The cloud-based LLM can then initiate a new round of planning (e.g., prompting the user, "Xiao Wang not found, can you provide his email address?"). In a preferred embodiment, the executed instruction is for handling complex tasks that require multiple iterative steps to complete collaboratively. In this case, the task processing instruction can be finally generated based on the server-side LLM's multiple planning processes and the intermediate results obtained from each plan.

[0081] Example 2: Figure 2 As shown, this disclosure provides an intelligent headphone interaction system based on a large language model intelligent assistant agent, including:

[0082] The interface registration module is configured to scan the tool interfaces provided by the local system to obtain the interface ID and interface name, find the interface description corresponding to the interface name, and combine the interface ID, interface name, and interface description to register them in the tool interface registry.

[0083] The Agent deployment module is configured to deploy a server-local architecture Large Language Model (LLM) service system, receive user registration applications, and the registration application information includes a tool interface registry. Based on the interface description in the tool interface registry, the tool interfaces are clustered by function, and the sub-intelligent assistant Agent of the Large Language Model (LLM) is set based on the function clustering results. The sub-intelligent assistant Agent is used to handle device operation transactions for the corresponding function.

[0084] The voice instruction processing module is configured to start a local background service of the large language model LLM when the intelligent earphone is connected to the local system, collect a voice instruction received by the intelligent earphone of the user, and send the voice instruction to the server end, the server end calls the large language model LLM to perform function analysis on the voice instruction, determines a function sequence to be executed for the voice instruction according to an analysis result, and calls a corresponding sub-intelligent assistant Agent according to a function type in the function sequence to perform processing.

[0085] The processing instruction execution module is configured to return a processing instruction generated by the sub-intelligent assistant Agent to the local end, and the local background service queries a tool interface corresponding to the processing instruction in a tool interface registry, and dynamically calls the tool interface through a code reflection mechanism to realize response of the user instruction.

[0086] In a preferred embodiment, the interface registration module specifically includes:

[0087] The first scanning unit is configured to determine a system version of a local operating system OS, find an official SDK API document, obtain an interface ID and an interface name based on the official SDK API document, and find an interface description corresponding to the interface name.

[0088] The second scanning unit is configured to scan APIs provided by local system applications and third-party applications through a Content Provider interface to obtain an interface ID and an interface name, find an interface description of a service document provided by the local system applications and the third-party applications based on the interface ID and the interface name, and find an interface description in the Internet based on the interface name.

[0089] The supplementary scanning unit is configured to perform a supplementary scanning step, obtain an interface ID and an interface name, and find an interface description corresponding to the interface name.

[0090] The registration unit is configured to combine the interface ID, the interface name, and the interface description into a triple (id, name, des) and register the triple in a tool interface registry.

[0091] In a preferred embodiment, the interface description at least includes permission level information and interface calling mode information of the interface, the permission level information and the interface calling mode information of the interface are combined with the interface ID, the interface name, and the interface description to form a quadruple (id, name, des, perm, schema), and the quadruple is registered in the tool interface registry.

[0092] In a preferred embodiment, the Agent deployment module specifically includes:

[0093] The function clustering unit is configured to convert interface descriptions of all tools in the tool interface registry into semantic vectors using an embedding model, and then determine function clusters by grouping tools with similar functions based on a clustering analysis model, and define a type for each function cluster;

[0094] The prompt word management unit is configured to write system prompt words for the sub-intelligent assistant Agent based on the type of the function cluster;

[0095] The assistant instantiation unit is configured to maintain system prompt words for the sub-intelligent assistant Agent, and generate an instance of the sub-intelligent assistant Agent based on a basic large language model (LLM) and the system prompt words, for processing device operation transactions of the corresponding function.

[0096] In a preferred embodiment, the voice instruction processing module specifically includes:

[0097] The understanding unit is configured to deeply understand the intent of the user voice instruction;

[0098] The function sequence generation unit is configured to generate a function sequence to be executed in combination with the understood intent;

[0099] The routing unit is configured to route the generated function sequence to the corresponding sub-intelligent assistant Agent, which analyzes the function sequence and generates a response instruction.

[0100] The above scheme at least achieves the following technical effects: (1) By the tool interface scanning and registration mechanism and the sub-intelligent assistant allocation scheme based on function clustering, the general large language model is constrained to multiple lightweight and specialized sub-intelligent assistant Agents, each of which only needs to process transactions in a specific function domain, and its instruction set and decision logic are greatly simplified, greatly reducing the dependence on cloud computing resources and significantly optimizing the control-response delay of the intelligent earphone, (2) The end-cloud collaboration architecture with cloud analysis and local execution ensures that private data does not leave the local area and authorized execution of sensitive functions, ensuring user privacy and building a security barrier. (3) The scanning and registration mechanism ensures the scalability of the intelligent earphone control function; it breaks through the limitations of the fixed and poor scalability of the voice assistant function of the existing mobile terminal, and solves the problem that it cannot automatically identify and utilize the functions of third-party applications installed on the user's device.

[0101] According to an embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium can have instructions (i.e., the above-mentioned elements implemented in software) that, when executed by a machine, cause the machine to perform the above-mentioned functions in the various embodiments of the present specification. Figure 1The various operations and functions described. Specifically, a system or apparatus equipped with a readable storage medium on which software program codes implementing the functions of any of the above-described embodiments are stored can be provided, and a computer or processor of the system or apparatus is caused to read out and execute the instructions stored in the readable storage medium.

[0102] In this case, the program codes read from the readable medium can implement the functions of any of the above-described embodiments by themselves, and thus the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present specification.

[0103] Embodiments of the readable storage medium include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as a CD-ROM, a CD-R, a CD-RW, a DVD-ROM, a DVD-RAM, a DVD- RW, a DVD-RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program codes can be downloaded from a server computer or a cloud over a communication network.

[0104] Those skilled in the art will understand that the various embodiments disclosed above can be modified and changed in various ways without departing from the spirit of the invention. Therefore, the scope of protection of the present specification should be defined by the appended claims.

[0105] It should be noted that not all steps and units in the above-described flowcharts and system block diagrams are necessary, and some steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in each of the above embodiments can be a physical structure or a logical structure, i.e., some units can be implemented by the same physical client, or some units can be implemented by multiple physical clients, or some units can be implemented by some components in multiple independent devices.

[0106] In each of the above embodiments, a hardware unit or module can be implemented mechanically or electronically. For example, a hardware unit, module or processor can include a permanent or temporary dedicated circuit or logic (such as a dedicated processor, FPGA or ASIC) to perform the corresponding operation. The hardware unit or processor can also include programmable logic or circuit (such as a general-purpose processor or other programmable processor) that can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical or dedicated permanent circuit, or temporarily configured circuit) can be determined based on cost and time considerations.

[0107] The detailed description set forth above describes exemplary embodiments but is not intended to represent the only embodiments in which the claimant can realize and use the present disclosure. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration," and not to mean "preferred" or "having some other perceived advantage." The detailed description includes specific details for the purpose of providing a thorough understanding of the technology described. However, it will be apparent to those skilled in the art that the technology described can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described embodiments.

[0108] The foregoing description of the present disclosure has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the disclosure be limited not with this detailed description, but rather by the claims appended hereto.

Claims

1. A smart headphone interaction method based on a large language model-based intelligent assistant agent, the method comprising the following steps: S100 scans the tool interfaces provided by the local system to obtain the interface ID and interface name, finds the interface description corresponding to the interface name, and combines the interface ID, interface name, and interface description and registers them in the tool interface registry. S110, Deploy a server-local architecture large language model LLM service system, receive user registration application information, the registration application information includes a tool interface registry; Based on the interface descriptions in the tool interface registry, the tool interfaces are clustered by function. Based on the function clustering results, a sub-intelligent assistant agent of the large language model LLM is set. The sub-intelligent assistant agent is used to handle device operation transactions for the corresponding functions. S120, when the smart earphone is connected to the local system, the local background service of the Large Language Model (LLM) is started. The local background service collects the voice commands received by the user's smart earphone and sends them to the server. The server calls the Large Language Model (LLM) to perform functional analysis on the voice commands, determines the function sequence to be executed for this voice command based on the analysis results, and calls the corresponding sub-intelligent assistant (Agent) to process the commands according to the function type in the function sequence. S130, the sub-intelligent assistant Agent returns the generated processing instructions to the local end. The local end background service queries the tool interface corresponding to the processing instructions in the tool interface registry, and dynamically calls the tool interface through code reflection mechanism to realize the response to the user instructions; Step S110 includes: S111, use an embedding model to convert the interface descriptions of all tools in the tool interface registry into semantic vectors, and then use a clustering analysis model to determine tools with similar combined functions to form functional clusters, and define a type for each functional cluster; S112, Based on the type of functional cluster, write system prompt words for the sub-intelligent assistant (Agent); S113 is for maintaining system prompt words for the sub-intelligent assistant agent. It generates sub-intelligent assistant agent instances based on the basic large language model LLM and system prompt words, which are used to handle device operation transactions for corresponding functions.

2. The method as described in claim 1, characterized in that: Step S100 includes: S101, determine the system version of the local operating system OS, find the official SDK API documentation, obtain the interface ID and interface name based on the official SDK API documentation, and find the interface description corresponding to the interface name. S102, scan the APIs provided by local system applications and third-party applications through the Content Provider interface to obtain the interface ID and interface name, and search for the interface description of the service documents provided by local system applications and third-party applications based on the interface ID and interface name. At the same time, search for the interface description on the Internet based on the interface name. S103, perform a supplementary scan step to obtain the interface ID and interface name, and find the interface description corresponding to the interface name; S104 combines the interface ID, interface name, and interface description into a triple (id, name, des) and registers it in the tool interface registry.

3. The method as described in claim 1, characterized in that, The step S120 specifically includes: S121, Deep Understanding of Intent; S122, Generate a functional sequence based on intent; S123, the function sequence is routed to the corresponding sub-intelligent assistant (Agent), the sub-intelligent assistant (Agent) analyzes the function sequence and generates response instructions.

4. A smart headphone interaction system based on a large language model intelligent assistant agent, characterized in that, The system includes: The interface registration module is used to scan the tool interfaces provided by the local system to obtain the interface ID and interface name, find the interface description corresponding to the interface name, and combine the interface ID, interface name, and interface description to register them in the tool interface registry. The Agent deployment module is used to deploy a server-local architecture Large Language Model (LLM) service system, receive user registration application information, which includes a tool interface registry; based on the interface description in the tool interface registry, the tool interfaces are clustered by function, and a sub-intelligent assistant Agent for the Large Language Model (LLM) is set based on the function clustering results. The sub-intelligent assistant Agent is used to handle device operation transactions for the corresponding function. The voice command processing module is used to start the local background service of the Large Language Model (LLM) when the smart earphone is connected to the local system. The local background service collects the voice commands received by the user's smart earphone and sends them to the server. The server calls the Large Language Model (LLM) to perform functional analysis on the voice commands, determines the function sequence to be executed for the current voice command based on the analysis results, and calls the corresponding sub-intelligent assistant (Agent) to process the commands according to the function type in the function sequence. The instruction execution module is used to return the processing instructions generated by the sub-intelligent assistant Agent to the local end. The local backend service queries the tool interface corresponding to the processing instruction in the tool interface registry and dynamically calls the tool interface through code reflection mechanism to realize the response to the user instruction. The Agent deployment module includes: Functional clustering units are used to convert the interface descriptions of all tools in the tool interface registry into semantic vectors using an embedding model. Then, based on the clustering analysis model, tools with similar combined functions are identified to form functional clusters, and a type is defined for each functional cluster. The prompt word management unit is used to write system prompt words for the sub-intelligent assistant (Agent) based on the type of function cluster. The assistant instantiation unit is used to maintain system prompt words for the sub-intelligent assistant (Agent), and to generate sub-intelligent assistant (Agent) instances based on the basic large language model (LLM) and system prompt words, which are used to handle device operation transactions for corresponding functions.

5. The system as described in claim 4, characterized in that, The interface registration module specifically includes: The first scanning unit is used to determine the system version of the local operating system (OS), search for the official SDK API documentation, obtain the interface ID and interface name based on the official SDK API documentation, and search for the interface description corresponding to the interface name. The second scanning unit is used to scan the APIs provided by local system applications and third-party applications through the Content Provider interface to obtain the interface ID and interface name, and to find the interface description of the service documents provided by local system applications and third-party applications based on the interface ID and interface name. At the same time, it searches for the interface description on the Internet based on the interface name. The supplementary scanning unit is used to perform supplementary scanning steps, obtain the interface ID and interface name, and find the interface description corresponding to the interface name. The registration unit is used to combine the interface ID, interface name, and interface description into a triple (id, name, des) and register it in the tool interface registry.

6. The system as described in claim 4, characterized in that, The voice command processing module specifically includes: The understanding unit is used to deeply understand the intent of the user's voice commands; Functional units are used to generate a sequence of functions to be executed by combining the understood intent; The routing unit is used to route the function sequence to the corresponding sub-intelligent assistant (Agent), which then analyzes the function sequence and generates response instructions.

Images