Tool invocation method, system and electronic device of end-side device

Abstract

The present disclosure relates to the technical field of artificial intelligence, and discloses a tool calling method and system of an end-side device and an electronic device. The end-side device registers a general device tool shared by multiple peripherals, and the tool takes a device identifier and an operation identifier as calling parameters; each peripheral corresponds to a device description file recording an interface protocol type, a device address, and an operation definition. The method comprises: receiving a tool call initiated by a large language model and carrying a device identifier of a target peripheral and an operation identifier; finding the corresponding device description file according to the device identifier, and parsing the interface protocol type, the device address, and the operation definition corresponding to the operation identifier therefrom; calling a driver corresponding to the interface protocol type to perform an input / output operation on the target peripheral; and returning the execution result to the large language model. The number of tools does not increase with the peripherals, and the addition of new peripherals only requires the addition of a device description file, thereby realizing the separation of hardware parameters and tool logic and the dynamic expansion of peripherals with zero code.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to tool invocation technology for edge devices. Background Technology

[0002] The descriptions in this section are intended to provide background information for the implementation of this disclosure and should not be construed as an admission or implication that they constitute prior art.

[0003] With the development of large language model technology, deploying intelligent agents on resource-constrained embedded edge devices is gradually becoming an industry trend. Edge agents understand user intent through large language models and operate various peripherals connected to the edge device through toolcall mechanisms, such as sensors, cameras, storage devices, and display devices connected via GPIO, I2C, SPI, and UART interfaces, thereby completing specific tasks.

[0004] In related technologies, edge-side intelligent agents typically adopt a method of registering an independent tool for each peripheral: each peripheral corresponds to an independent tool processing function and its parameter pattern description, and the hardware parameters of the peripheral are written in the code of the tool processing function; at runtime, the complete parameter pattern descriptions of all tools are injected into the context of a large language model, from which the large language model selects tools and fills in the calling parameters. Under this approach, adding or replacing peripherals requires writing driver code, defining parameter pattern descriptions, modifying the tool registry, and recompiling the firmware, making the expansion process complex and difficult to complete at runtime; at the same time, as the types of peripherals increase, the number of tools grows linearly, and the content injected into the context of the large language model continues to expand, generating a large amount of token overhead, which increases the operational burden of the resource-constrained edge-side environment. Summary of the Invention

[0005] This disclosure provides a tool invocation method, system, and electronic device for edge devices. One technical problem it can solve is how to reduce the coupling between peripherals and tools in edge devices and simplify the implementation of operating multiple peripherals through tool invocation of a large language model.

[0006] According to one aspect of this disclosure, a tool invocation method for an end-side device is provided, applied to an end-side device connected to multiple peripherals. The end-side device has a registered general-purpose device tool, which uses a device identifier and an operation identifier as invocation parameters. The multiple peripherals share the general-purpose device tool. Each of the multiple peripherals corresponds to a device description file, which records the hardware parameters of the corresponding peripheral in a structured format. The hardware parameters include an interface protocol type, a device address, and an operation definition. The method includes: The system receives a tool call to the general-purpose device tool initiated by the large language model in response to a user request. The tool call carries the device identifier of the target peripheral and the operation identifier of the operation to be performed. Based on the device identifier carried by the tool call, the device description file corresponding to the target peripheral is searched, and the interface protocol type, device address, and operation definition corresponding to the operation identifier carried by the tool call are parsed from the found device description file. The driver corresponding to the interface protocol type is invoked, and the driver performs input / output operations on the target peripheral according to the device address and the operation definition. The results of the input / output operations are returned to the large language model.

[0007] In a preferred embodiment, the method further includes, before receiving the tool invocation: In response to the user request, without invoking the large language model, the local classifier running on the edge device filters out candidates that match the user request from the candidate set. The number of candidates filtered out does not exceed a preset value, and the candidate set includes at least the general device tool. The name and function description of each selected candidate are sent to the large language model, and the content sent does not include the parameter pattern description, which records the name, type and value constraints of the corresponding tool's call parameters; The system receives the selection result returned by the large language model for the selected candidates, the selection result indicating the selected target candidate, and injects the parameter pattern description corresponding to the target candidate into the context of the large language model, so that the large language model generates the tool call carrying the call parameters based on the injected parameter pattern description.

[0008] In a preferred embodiment, the local classifier uses at least one of keyword matching, rule engine, word frequency statistics model or vector similarity model to determine items that match the user request from the candidate set, and uses the determined items as the filtered candidates.

[0009] In a preferred embodiment, the operation definition includes a read operation definition, which records the target register address, the read data length, and the decoding formula; the device description file also records an initialization sequence, which includes at least one ordered register write operation item, each register write operation item recording the register address, the data to be written, and the delay parameter; Performing input / output operations on the target peripheral includes: initializing the target peripheral according to the initialization sequence, and reading raw data from the target peripheral according to the target register address and the read data length; The method further includes: before returning the execution result to the large language model, the general-purpose device tool decodes the original data into physical quantity values ​​according to the decoding formula, and uses the physical quantity values ​​as the execution result.

[0010] In a preferred embodiment, the general-purpose device tools include a device reading tool, a device writing tool, and a device control tool; The device reading tool is used to read data from the target peripheral device; The device writing tool is used to write data or configuration parameters to the target peripheral device; The device control tool is used to control the target peripheral to perform actions according to the control sequence recorded in the device description file corresponding to the target peripheral.

[0011] In a preferred embodiment, the method further includes: During the operation of the terminal device, the device description file corresponding to the newly added peripheral is loaded from the file system of the terminal device; The loaded device description file is parsed, and the device identifier recorded in the loaded device description file is registered as an available value of the device identifier in the calling parameters of the general device tool.

[0012] In a preferred embodiment, the candidate set further includes at least one skill item, each skill item being defined by a skill description file, which records triggering conditions, execution step sequence and parameter constraint rules, but does not record any peripheral hardware parameters; When the target candidate is a skill item, the parameter pattern description corresponding to the target candidate includes the parameter pattern description of the general equipment and tools involved in the execution step sequence; and The method further includes: sequentially receiving and executing tool calls to the general device tool initiated by the large language model according to the execution step sequence.

[0013] According to another aspect of this disclosure, a tool invocation system for an end-side device is provided, deployed on an end-side device connected to multiple peripherals, the system including a device configuration layer, a general device tool layer, and a driver execution layer; The device configuration layer stores a device description file for each of the plurality of peripherals. The device description file records the hardware parameters of the corresponding peripheral in a structured format. The hardware parameters include interface protocol type, device address, and operation definition. The general device tool layer registers general device tools, which use device identifiers and operation identifiers as invocation parameters. Multiple peripherals share these general device tools. The general device tool layer receives tool invocations initiated by the large language model in response to user requests. These tool invocations carry the device identifier of the target peripheral and the operation identifier of the operation to be performed. The general device tool layer also searches the device configuration layer for the target peripheral based on the device identifier carried in the tool invocation. It parses the interface protocol type, device address, and operation definition corresponding to the operation identifier carried in the tool invocation from the found device description file, and then invokes the driver corresponding to the interface protocol type in the driver execution layer. The driver execution layer includes drivers categorized by interface protocol type. The invoked driver performs input / output operations on the target peripheral according to the device address and the operation definition, and returns the execution result of the input / output operations to the general device tool layer. The general-purpose device tool layer is also used to return the execution result to the large language model.

[0014] According to another aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the methods described above.

[0015] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the method described above.

[0016] In the embodiments of this disclosure, a general-purpose device tool shared by multiple peripherals is registered in the end-side device, using device identifier and operation identifier as calling parameters. Each peripheral is configured with a device description file that records hardware parameters such as interface protocol type, device address, and operation definition in a structured format. After receiving a tool call initiated by the large language model, the device tool searches for the target peripheral's device description file based on the device identifier it carries, parses out the interface protocol type, device address, and operation definition corresponding to the operation identifier, and then calls the driver corresponding to the interface protocol type to perform input / output operations on the target peripheral and returns the execution result to the large language model. This ensures that the number of tools does not increase with the number of peripherals, the tool interface faced by the large language model remains stable, the number of tool descriptions injected into the large language model context is reduced, and token overhead is lowered. At the same time, the hardware parameters of the peripherals are separated from the tool execution logic. Adding or replacing peripherals only requires modifying the corresponding device description file, without modifying the tool code or tool registry, realizing zero-code dynamic expansion of peripherals and reducing the cost of accessing and maintaining multiple peripherals on the end-side device.

[0017] Furthermore, by having a local classifier on the device first filter out a limited number of candidates from the candidate set without calling the large language model, and only sending the name and functional description of the candidate to the large language model without sending the parameter pattern description, and then injecting the parameter pattern description into the context of the large language model after the large language model selects the target candidate, the number of tokens injected into the context of the large language model in each round can be significantly reduced, the model calling cost and the waiting time before inference can be reduced, and the filtering process is completed locally on the device without introducing additional model interaction delay.

[0018] Furthermore, by employing at least one of keyword matching, rule engine, word frequency statistics model, or vector similarity model to implement a local classifier, candidate selection can be completed quickly on the edge with extremely low computational and memory overhead.

[0019] Furthermore, by recording the read operation definition, which includes the target register address, the length of the read data, and the decoding formula, as well as the initialization sequence consisting of ordered register write operations in the device description file, and having the general-purpose device tool decode the read raw data into physical quantity values ​​according to the decoding formula and return it, the register-level operation details and data decoding logic can be completely carried by the device description file. The large language model does not need to understand the register mapping of the peripheral device to obtain directly usable physical quantity results, improving the accuracy and ease of use of the call.

[0020] Furthermore, by dividing general-purpose device tools into device reading tools, device writing tools, and device control tools, a constant small number of tools can cover operation types such as data reading, data writing, and motion control of peripheral devices, balancing the simplicity of tool interfaces with the completeness of operational capabilities.

[0021] Furthermore, by loading and parsing the device description file of the newly added peripheral from the file system during the operation of the end device, and registering the device identifier recorded therein as an available value for the general device tool call parameter, dynamic access of the peripheral at runtime can be achieved without modifying the code or recompiling the firmware.

[0022] Furthermore, by adding skill items defined by skill description files without hardware parameters to the candidate set, and injecting parameter pattern descriptions of the general equipment tools involved in the execution step sequence when a skill item is selected, and sequentially executing tool calls initiated by the large language model according to the execution step sequence, multi-step business orchestration can be achieved on top of general equipment tools, enabling the tool granularity to be flexibly adjusted at different abstraction levels, and the skill layer and hardware parameters are completely decoupled.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Various aspects and embodiments of this disclosure can be combined in different ways. This part of the disclosure is not intended to identify key or essential features of the claimed subject matter, nor is it intended to aid in determining the scope of the claimed subject matter. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a tool invocation method for an end-side device according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a two-stage tool selection mechanism according to an embodiment of the present disclosure. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this disclosure clearer, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Example 1 This embodiment provides a tool invocation method for an edge device, applicable to an edge device connected to multiple peripherals. The edge device can be, for example, an embedded device running a real-time operating system or a simplified Linux system. Peripherals refer to functional devices connected to the edge device via interfaces such as GPIO, I2C, SPI, UART, PCIe, or MIPI, such as temperature, humidity, and pressure sensors, image sensors, storage devices, and display devices. User requests can be input by the user in the form of voice or text to the edge device. The edge device understands the user request using a large language model and completes the corresponding task by operating the peripherals. The large language model can be deployed on the cloud and invoked by the edge device through a network interface, or it can be a lightweight large language model deployed locally on the edge device; this embodiment does not limit this.

[0027] In this embodiment, a general-purpose device tool is registered in the end-side device. This general-purpose device tool uses a device identifier and an operation identifier as its calling parameters, and multiple peripherals connected to the end-side device share this general-purpose device tool. That is, unlike the approach of registering a dedicated tool for each peripheral separately, in this embodiment, regardless of the number of peripherals connected to the end-side device, the large language model operates on the peripherals through the shared general-purpose device tool. The differences between different peripherals are distinguished by the device identifier in the calling parameters, thus the number of registered tools does not increase with the number of peripherals. This embodiment uses the registration of a general-purpose device tool named `device_execute` as an example. The calling parameters of this tool include the device identifier `device_name` and the operation identifier `operation`. Additionally, it can carry additional parameters related to specific operations as needed.

[0028] Each peripheral in a multi-device configuration corresponds to a device description file. The device description file records the hardware parameters of the corresponding peripheral in a structured format. These parameters include the interface protocol type, device address, and operation definition. The interface protocol type indicates the interface or bus type used by the peripheral, such as I2C, SPI, UART, GPIO, PCIE, or MIPI. The device address is used to address the peripheral on the corresponding bus, such as the slave address on an I2C bus. The operation definition describes the various operations supported by the peripheral and the input / output methods when performing these operations. The specific content of the operation definition can be flexibly set according to the peripheral type. For example, for register-type peripherals, the operation definition can record the register addresses and data length or data format involved in the operation. This embodiment uses a JSON format device description file as an example. Each peripheral corresponds to an independent JSON file, and the size of a single file is typically 0.5KB to 5KB. As an example, a temperature, humidity, and barometric pressure sensor is connected to an end-side device via an I2C bus. Its device identifier is env_sensor, and its I2C slave address is 0x76. The device description file for this peripheral can adopt the following structure: { "device_name": "env_sensor", "description": "temperature, humidity and pressure sensor", "protocol": "i2c", "bus_id": 0, "address": "0x76", "operations": [{"name": "temperature", "type": "read", "register": "0xF7", "length": 3},{"name": "humidity", "type": "read", "register": "0xFC", "length": 2},{"name": "set_mode", "type": "write", "register": "0xF4", "data_format": "uint8", "valid_range": [0, 255]} ]} Device description files can be packaged into the flash memory of the device during the firmware compilation stage, or they can be stored in the file system of the device and loaded at runtime. In addition to JSON, structured formats can also use text formats such as YAML and TOML, or binary encoding formats such as CBOR. Those skilled in the art can choose according to the trade-off between readability and storage overhead.

[0029] like Figure 1 As shown, the method includes steps 101 to 104, which will be described one by one below.

[0030] Step 101: Receive the tool call to the general device tool initiated by the large language model in response to the user request. The tool call carries the device identifier of the target peripheral and the operation identifier of the operation to be performed.

[0031] Specifically, taking a user's voice input of "What is the indoor temperature?" as an example, the edge device provides this user request to the large language model. After understanding the user's intent, the large language model initiates a tool call to the general device tool `device_execute`. The tool call can be given in structured JSON format, where the device identifier `device_name` is set to `env_sensor`, and the operation identifier `operation` is set to `temperature`, indicating that the temperature needs to be read from the target peripheral device identified as `env_sensor`.

[0032] Step 102: Based on the device identifier carried in the tool call, locate the device description file corresponding to the target peripheral, and parse the interface protocol type, device address, and operation definition corresponding to the operation identifier carried in the tool call from the located device description file.

[0033] Specifically, the edge device uses the device identifier as an index to locate the device description file corresponding to `env_sensor` in the storage location of the device description file, and then parses the file. Parsing can be performed by a lightweight JSON parser, which uses only stack memory, has a runtime memory footprint of less than 2KB, and does not perform dynamic memory allocation, making it suitable for memory-constrained edge environments. Through parsing, the target peripheral's interface protocol type (I2C), bus number (0), device address (0x76), and the operation definition corresponding to the operation identifier `temperature` are obtained. This operation definition indicates that this is a read operation, requiring the reading of 3 bytes of data starting from register address 0xF7.

[0034] Step 103: Invoke the driver corresponding to the interface protocol type. The driver performs input / output operations on the target peripheral according to the device address and operation definition.

[0035] Specifically, the end-side devices are equipped with drivers categorized by interface protocol type, such as I2C drivers, SPI drivers, UART drivers, GPIO drivers, PCIe drivers, and MIPI drivers. Each type of driver provides standardized read, write, and control interfaces. The driver does not contain any specific peripheral's business logic, nor is it aware of whether a peripheral is a temperature sensor or a barometer; it is only responsible for performing register-level input / output operations on a specified bus at a specified device address. The business logic related to the peripheral is entirely carried out by the device description file. In this example, the general-purpose device tool calls the I2C driver based on the I2C interface protocol type. The I2C driver initiates a read operation on bus 0 to slave address 0x76, reading 3 bytes of raw data starting from register address 0xF7 according to the operation definition.

[0036] Step 104: Return the execution results of the input and output operations to the large language model.

[0037] Specifically, the general-purpose device tool returns the execution results of the input / output operations to the large language model. The large language model then generates and outputs a user-oriented natural language response based on the execution results, such as "The current indoor temperature is 25.3 degrees Celsius". The execution results can be the data itself, the result of data processing, or status information indicating whether the operation was successful or failed.

[0038] Through steps 101 to 104 above, the large language model only needs to understand and call a constant number of general-purpose device tools. The differences between peripherals are degraded to the device description file, thereby decoupling the peripheral perception capability from the large language model tool calls. On the one hand, the number of registered tools does not increase linearly with the number of peripherals, avoiding the expansion of the tool list caused by the increase in the types of peripherals; on the other hand, hardware parameters and tool execution logic are completely separated. When replacing peripherals of the same type but with different parameters, such as changing the I2C slave address of the same type of sensor from 0x76 to 0x77, only the device address field in the device description file needs to be modified, without modifying any tool logic code or recompiling the firmware.

[0039] The above constitutes a complete technical solution, which enables those skilled in the art to invoke tools on the terminal device. Based on this, several optional improvements to this embodiment are described below.

[0040] In some optional implementations, the operation definition may include a read operation definition, which records the target register address, the length of the read data, and the decoding formula. The device description file may also record an initialization sequence, which includes at least one ordered register write operation item. Each register write operation item records the register address, the data to be written, and a delay parameter. Taking the temperature, humidity, and pressure sensor mentioned above as an example, its device description file can add an initialization sequence field, `init_sequence`, which contains a register write operation item. This item records a register address of 0xF4, a data to be written of 0x2C, and a delay parameter of 10 milliseconds, indicating that after writing 0x2C to register 0xF4, there is a 10-millisecond delay. The read operation definition for temperature, in addition to recording the target register address 0xF7 and the 3-byte read data length, further records the decoding formula and the output unit. At this point, performing input / output operations on the target peripheral includes: initializing the target peripheral according to the initialization sequence, that is, executing each register write operation item sequentially, and delaying according to the corresponding delay parameter after each write; then reading the raw data from the target peripheral according to the target register address and the data length; and before returning the execution result to the large language model, the general-purpose device tool decodes the raw data into a physical quantity value according to the decoding formula, and uses this physical quantity value as the execution result. Taking temperature reading as an example, the decoding process can be represented as follows: in, This represents an unsigned integer consisting of the high 16 bits of the raw data read from the target peripheral (i.e., the 3 bytes of raw data are concatenated byte by byte and then right-shifted by 8 bits to obtain the high 16 bits). That is, 65536 is the normalization constant. This represents the decoded physical quantity value, i.e., the temperature value, in degrees Celsius. Similarly, the read operation definition for humidity can record the target register address as 0xFC, the data length as 2 bytes, and the decoding formula as follows: ,in This represents the relative humidity value obtained from decoding, in percentage form. The meaning is the same as before. Using the above method, register addressing, initialization timing, and data decoding are all driven by the device description file and automatically completed by general-purpose device tools. Large language models do not need to understand the register mappings of peripherals to directly obtain numerical values ​​with clear physical meanings, which helps reduce the understanding burden of large language models and improve the accuracy of parameter input and inference. It should be noted that the initialization sequence and decoding formula are not necessary conditions for implementing this method. For peripherals that do not require initialization, the initialization sequence can be omitted. For operations that do not require numerical conversion, the original data can be directly returned as the execution result, and this method will still run normally in these cases.

[0041] As a preferred but not mandatory improvement, general-purpose device tools may include device reading tools, device writing tools, and device control tools, for example, named device_read, device_write, and device_control, respectively. The device reading tool is used to read data from the target peripheral, and its execution process is the same as the reading process shown in steps 102 to 104 above. The device writing tool is used to write data or configuration parameters to the target peripheral, for example, writing the working mode configuration value to register 0xF4 according to the write operation definition recorded in the device description file. Before writing, the call parameters can also be verified according to the data format and value range recorded in the write operation definition. The device control tool is used to control the target peripheral to perform actions according to the control sequence recorded in the device description file corresponding to the target peripheral, for example, completing the power-on initialization of the image sensor or completing the mounting of the storage device according to the control sequence. The control sequence can be recorded in the device description file as a type of operation definition. The control sequence includes at least one ordered operation item. Each operation item, for example, records a register address, write data, and delay parameters. Its structure can be the same as the register write operation item in the aforementioned initialization sequence, thereby reusing the same parsing and execution logic. Dividing responsibilities into three general-purpose device tools—read, write, and control—makes the calling parameters of each tool simpler and facilitates accurate parameter input for large language models. Of course, besides this three-tool division, as shown above, they can also be merged into a single general-purpose device tool and differentiated by operation identifiers or additional parameters. Alternatively, they can be subdivided into several general-purpose device tools according to the peripheral's interface protocol category. Those skilled in the art can flexibly choose based on the type of peripheral and operational complexity. The characteristic of multiple peripherals sharing a general-purpose device tool remains unchanged under all the above division methods.

[0042] Building upon the aforementioned solution, further support for dynamic runtime expansion of peripherals can be provided. Specifically, during the operation of the edge device, the device description file corresponding to the newly added peripheral is loaded from the edge device's file system. The loaded device description file is parsed, and the device identifier recorded in the loaded device description file is registered as an available value for the call parameters of the general device tool. When a new peripheral needs to be connected, such as adding a barometric pressure sensor, the developer or user only needs to write the device description file for the peripheral according to the agreed format specifications and put it into the edge device's file system. After the edge device detects the new device description file during operation, it automatically completes the parsing and registration, adding the device identifier recorded therein to the list of available device identifier values ​​for the general device tool. In the next round of large language model requests, the large language model can operate the newly added peripheral by specifying the device identifier in the call parameters. Since the drivers for each interface protocol are pre-installed in the edge device according to the protocol type, the entire access process requires no code writing, no modification to the tool registry, no recompilation of firmware, and no online upgrade of the entire device, thus achieving zero-code dynamic expansion of peripherals. It should be noted that dynamic loading at runtime is not a necessary condition for implementing this method. The device description file can also be statically packaged during the firmware compilation stage, which can also achieve the separation of hardware parameters and tool logic.

[0043] Example 2 This embodiment, building upon Embodiment 1, further provides a two-stage tool selection mechanism. This mechanism determines the tool information to be provided to the large language model before receiving tool calls, thereby reducing the token overhead of each round of large language model calls. In a client-side scenario, the total number of candidate tools and skills can reach 20 to 200. If the complete parameter pattern descriptions of all candidates are injected into the context of the large language model at once, a single request may consume 5,000 to 15,000 or more tokens, increasing both the call cost of the large language model and the waiting time before inference. Therefore, this embodiment first performs a coarse screening by the local classifier on the client side without calling the large language model, and then the large language model selects from a small number of candidates and injects parameter pattern descriptions as needed.

[0044] In this embodiment, the parameter schema description records the name, type, and value constraints of the calling parameters for the corresponding tool. For example, the parameter schema description can follow the JSON Schema specification, recording the name, data type, value range or enumeration value, whether it is required, and validation rules for each calling parameter.

[0045] like Figure 2 As shown, the two-stage tool selection mechanism includes steps 201 to 203, which are executed before step 101 in Embodiment 1.

[0046] Step 201: In response to the user request, without invoking the large language model, the local classifier running on the device side is used to filter out candidates that match the user request from the candidate set. The number of candidates filtered out does not exceed a preset value, and the candidate set includes at least general device tools.

[0047] Specifically, this embodiment uses a local classifier employing keyword matching as an example. Several keywords are pre-labeled for each candidate in the candidate set, and an inverted index table from keywords to candidate options is constructed. The local classifier segments the text requested by the user, queries the inverted index table based on the segmentation results, counts the number of keywords matched for each candidate option, calculates the matching degree accordingly, and selects candidates from high to low matching degrees, with the selected number not exceeding a preset value. The preset value can be configured according to the resource status of the edge device, typically ranging from 3 to 8. The local classifier runs on the microcontroller of the edge device. When using keyword matching or rule matching, its memory usage can be controlled within 10KB, and the execution time of a single filtering is less than 1 millisecond. The entire filtering process does not require calling a large language model, thus not introducing additional network latency.

[0048] Step 202: Send the name and functional description of each selected candidate to the large language model, and the content sent does not include the parameter pattern description.

[0049] Specifically, the functional description is a concise explanation of the candidate's function; for example, the functional description of each candidate should not exceed 80 characters. Since the content sent to the large language model at this stage only includes the name and brief functional description of each candidate, and does not include parameter pattern descriptions, the size of the content injected into the large language model context is significantly compressed.

[0050] Step 203: Receive the selection result returned by the large language model for the selected candidates. The selection result indicates the selected target candidate. Inject the parameter pattern description corresponding to the target candidate into the context of the large language model so that the large language model can generate a tool call carrying the calling parameters based on the injected parameter pattern description.

[0051] Specifically, the large language model selects one or more target candidates corresponding to the user's request from the filtered candidates and returns the selection result; based on the selection result, the edge device injects only the parameter pattern description corresponding to the target candidate into the context of the large language model, and the large language model fills in the precise calling parameters accordingly and initiates the tool call. Thereafter, the edge device receives and executes the tool call according to steps 101 to 104 of Embodiment 1.

[0052] By adopting the two-stage mechanism described above, the number of tool-related content injected in each round of large language model invocation can be reduced from over 5,000 to 15,000 tokens under the full injection method to 400 to 800 tokens, reducing token overhead by more than 80%. Simultaneously, the first-stage coarse screening is completed locally on the client side, taking less than 1 millisecond and requiring no involvement of the large language model. Compared to the method where the large language model actively queries the tool list through additional interaction rounds to obtain complete descriptions, this reduces the number of interaction rounds for the large language model and the corresponding response latency.

[0053] In some optional implementations, the local classifier can employ at least one of the following methods besides keyword matching: a rule engine, a word frequency statistical model, or a vector similarity model. These methods can also be combined. The rule engine matches user requests using pre-configured conditions and action rules, outputting candidate options associated with a rule when a rule is matched. The word frequency statistical model, for example, can use a TF-IDF model, encoding the functional description of each candidate option and the user request separately using TF-IDF, calculating the similarity between their encoded vectors, and selecting candidates based on similarity. This method eliminates the need for manual keyword annotation. The vector similarity model, for example, can use a small semantic vector model obtained through knowledge distillation, encoding the user request and the functional description of each candidate option into semantic vectors before calculating similarity. This model has a stronger understanding of synonyms and semantic variations, and its size is approximately 100KB to 300KB, making it suitable for edge devices with relatively ample memory. When using a word frequency statistical model or a vector similarity model, the similarity can be calculated using cosine similarity. in, This represents the encoded vector of the user request. An encoding vector representing the functional description of a candidate option. Represents the dot product of two vectors. and Let these represent the magnitudes of the two vectors, Indicates similarity. A larger value indicates a closer match between the candidate and the user's request. Those skilled in the art can flexibly choose from the various implementation methods described above based on the memory resources of the edge device, the size of the candidate options, and the requirements for semantic understanding accuracy.

[0054] In some optional implementations, the candidate set may include at least one skill item in addition to general-purpose device tools. Each skill item is defined by a skill description file, which records the triggering conditions, execution step sequence, and parameter constraint rules, but does not record any peripheral hardware parameters. The skill description file can use a purely descriptive format such as Markdown, JSON, or YAML. Taking an environmental monitoring skill item as an example, its skill description file records the triggering condition as a user request to query environmental conditions, the execution step sequence as first reading the temperature through a general-purpose device tool, then reading the humidity, and finally outputting the results to a display device, and the parameter constraint rules as, for example, limiting the device identifier of the target peripheral for temperature and humidity reading and the numerical format of the output results. When the target candidate selected in step 203 is a skill item, the parameter mode description corresponding to the target candidate includes the parameter mode description of the general-purpose device tool involved in the execution step sequence of the skill item; subsequently, the edge device sequentially receives and executes each tool call to the general-purpose device tool initiated by the large language model according to the execution step sequence, thereby completing the multi-step task defined by the skill item. The granularity of skill items can be flexibly set. Coarse-grained skill items can be used to schedule multiple tool calls, while fine-grained skill items can correspond to only a single tool call. Since skill description files do not record hardware parameters or contain program code, adding a new skill only requires adding a new skill description file, and no code modification is required. It should be noted that skill items are not a necessary condition for implementing this method. When the candidate set only contains general-purpose equipment tools, the above two-stage tool selection mechanism can also be implemented.

[0055] Example 3 This embodiment provides a tool invocation system for an end-side device, which can be used to implement the methods of Embodiment 1 or Embodiment 2 described above. The system is deployed on an end-side device connected to multiple peripherals. The system includes a device configuration layer, a general device tool layer, and a driver execution layer.

[0056] The device configuration layer stores a device description file for each of the multiple peripherals. Each device description file records the hardware parameters of the corresponding peripheral in a structured format, including the interface protocol type, device address, and operation definition. Specifically, the device configuration layer can be implemented by a file system partition in the flash memory of the end device. Each peripheral corresponds to an independent structured file, and the file format and content are the same as described in Embodiment 1, and will not be repeated here. The device configuration layer provides file lookup and retrieval capabilities indexed by the device identifier.

[0057] The general device tool layer registers general device tools, which use device identifiers and operation identifiers as invocation parameters. Multiple peripherals share the same general device tool. The general device tool layer receives tool calls initiated by the large language model in response to user requests. These tool calls carry the device identifier of the target peripheral and the operation identifier of the operation to be performed. The general device tool layer also searches the device configuration layer for the corresponding device description file based on the device identifier carried in the tool call. From the found device description file, it parses the target peripheral's interface protocol type, device address, and operation definition corresponding to the operation identifier carried in the tool call, and then invokes the driver corresponding to that interface protocol type in the driver execution layer.

[0058] The driver execution layer includes drivers categorized by interface protocol type, such as GPIO drivers, I2C drivers, SPI drivers, UART drivers, PCIe drivers, and MIPI drivers. Each type of driver provides standardized read, write, and control interfaces. The invoked driver performs input / output operations on the target peripheral based on the device address and operation definition, and returns the results of the input / output operations to the general device tool layer. The driver execution layer does not contain any specific peripheral's business logic; it only handles register-level physical input / output. The business differences between peripherals are orchestrated by the general device tool layer based on the device description file.

[0059] The general-purpose device tool layer is also used to return the execution results to the large language model. Thus, the three-layer structure of the system has a clear division of labor: the device configuration layer carries hardware parameters, the general-purpose device tool layer carries tool abstraction and execution orchestration, and the driver execution layer carries physical input and output. The layers interact through standardized interfaces, and changes in one layer do not affect other layers. For example, replacing peripherals or adjusting their hardware parameters only involves updating files in the device configuration layer, and supporting new interface protocols only involves extending drivers in the driver execution layer; neither requires modifying the logic of the general-purpose device tool layer.

[0060] Based on the aforementioned three-layer structure, the system may further include at least one of the following: a skill layer, a two-stage scheduling engine, an operating system abstraction layer, and a user interaction layer. The skill layer defines multi-step tool orchestration rules and constraints using purely descriptive files, and its implementation is the same as the description of skill items in Embodiment 2. The two-stage scheduling engine includes a local classifier running on the edge device, used to execute the two-stage tool selection mechanism described in Embodiment 2. The operating system abstraction layer encapsulates operating system primitives such as thread creation and management, mutexes, semaphores, message queues, software timers, memory allocation and release, hierarchical logging, and system time. Each layer in the system calls system services through the interface of the operating system abstraction layer, without directly relying on the interface of any specific real-time operating system, thus facilitating the portability of the system between different real-time operating systems or simplified Linux systems. The user interaction layer receives user requests in voice or text form and outputs responses to the user through speech synthesis or screen display. Optionally, the execution of tool calls in the general device tool layer can be scheduled across threads using a circular queue. The depth of the circular queue can be set to 8 to 16 slots. When the queue is full, backpressure control is applied to newly arriving tool calls to ensure the stability of the execution process under resource-constrained environments. It should be noted that the skill layer, two-stage scheduling engine, operating system abstraction layer, and user interaction layer are not necessary components of this system. The system consisting only of the device configuration layer, general device tool layer, and driver execution layer can fully realize the tool invocation of the end device.

[0061] Example 4 This embodiment provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the methods described in Embodiment 1 or Embodiment 2. Specifically, the electronic device can be an embedded edge device. The processor can be a microcontroller or an embedded microprocessor with a clock frequency of tens to hundreds of megahertz. The memory can include random access memory and flash memory, wherein the capacity of the random access memory is, for example, 512KB to 16MB, and the capacity of the flash memory is, for example, 2MB to 16MB. The flash memory stores the aforementioned computer program and device description files for each peripheral. The electronic device may also include interface circuits for connecting to peripherals, such as one or more of a GPIO controller, I2C controller, SPI controller, UART controller, PCIe controller, and MIPI controller, as well as a network interface for communicating with a large language model on the cloud side. A real-time operating system or a simplified Linux system can run on the processor.

[0062] Furthermore, one embodiment of this disclosure provides a computer-readable storage medium storing computer-executable instructions (or a computer program) that, when executed by a processor, implement the steps in the above-described method embodiments. The computer-readable storage medium may include any type of volatile or non-volatile memory, or any combination thereof, storing computer program code. Specifically, the computer-readable storage medium may include, but is not limited to: magnetic storage devices (e.g., magnetic tape, hard disk, floppy disk), optical storage devices (e.g., optical disc (CD), digital versatile optical disc (DVD)), magneto-optical storage devices (e.g., magneto-optical disc), semiconductor memory (e.g., read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc.) or other storage technologies. The computer-readable storage medium in this disclosure refers to a non-transitory readable storage medium, excluding the transient propagation signal itself (e.g., modulated data signals, carrier waves, etc.).

[0063] Furthermore, one embodiment of this disclosure provides a computer program product including a computer program or computer-executable instructions carried on a non-transitory computer-readable medium, wherein the computer program or computer-executable instructions, when executed by a processor, cause the processor to perform the steps in the above-described method embodiments.

[0064] It should be noted that the terms "first," "second," etc., used in this specification are only used to distinguish similar objects, and the elements defined by them do not necessarily require or imply any actual relationship or order between these elements, nor are they used to describe a specific order or sequence.

[0065] The terms “comprising,” “including,” or any other variations thereof, as used herein, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] The phrase "execute according to a certain element" as used in this specification means at least according to that element, including both "execute only according to that element" and "execute according to that element and other elements".

[0067] Those skilled in the art will understand that variations of the method steps described above can be obtained based on the embodiments of this disclosure without any inventive effort. For example, the sequence numbers of the steps described in the method embodiments of this specification do not themselves constitute a limitation on the execution order of these steps. Unless there is an explicit specific limitation in the context, the steps may be executed in a different order than that shown in the embodiments (e.g., executing the steps with larger sequence numbers first, then the steps with smaller sequence numbers), or they may be executed in parallel. Furthermore, other steps may be inserted between multiple steps with consecutively numbered sequence numbers.

[0068] Those skilled in the art will also understand that the division of modules and units described in the embodiments of this disclosure is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. The modules or units may or may not be physically separate. The components shown as modules or units may or may not be physical modules. They may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the embodiments of this disclosure according to actual needs. In addition, the connection or interaction relationships shown in the accompanying drawings are only illustrative and do not constitute a limitation of this disclosure.

[0069] Furthermore, it should be understood that after reading the contents of this disclosure, those skilled in the art can make various alterations or modifications to this disclosure, and these equivalent forms also fall within the scope of protection claimed by this disclosure.

Claims

1. A method for invoking tools on an end-side device, characterized in that, The method is applied to an end-side device connected to multiple peripherals. The end-side device has a registered general-purpose device tool, which uses a device identifier and an operation identifier as calling parameters. The multiple peripherals share this general-purpose device tool. Each of the multiple peripherals corresponds to a device description file, which records the hardware parameters of the corresponding peripheral in a structured format. These hardware parameters include the interface protocol type, device address, and operation definition. The method includes: The system receives a tool call to the general-purpose device tool initiated by the large language model in response to a user request. The tool call carries the device identifier of the target peripheral and the operation identifier of the operation to be performed. Based on the device identifier carried by the tool call, the device description file corresponding to the target peripheral is searched, and the interface protocol type, device address, and operation definition corresponding to the operation identifier carried by the tool call are parsed from the found device description file. The driver corresponding to the interface protocol type is invoked, and the driver performs input / output operations on the target peripheral according to the device address and the operation definition. The results of the input / output operations are returned to the large language model.

2. The method according to claim 1, characterized in that, Before receiving the tool call, the method further includes: In response to the user request, without invoking the large language model, the local classifier running on the edge device filters out candidates that match the user request from the candidate set. The number of candidates filtered out does not exceed a preset value, and the candidate set includes at least the general device tool. The name and function description of each selected candidate are sent to the large language model, and the content sent does not include the parameter pattern description, which records the name, type and value constraints of the corresponding tool's call parameters; The system receives the selection result returned by the large language model for the selected candidates, the selection result indicating the selected target candidate, and injects the parameter pattern description corresponding to the target candidate into the context of the large language model, so that the large language model generates the tool call carrying the call parameters based on the injected parameter pattern description.

3. The method according to claim 2, characterized in that, The local classifier uses at least one of keyword matching, rule engine, word frequency statistics model or vector similarity model to determine the items that match the user request from the candidate set, and uses the determined items as the filtered candidates.

4. The method according to claim 1, characterized in that, The operation definition includes a read operation definition, which records the target register address, the length of the read data, and the decoding formula; the device description file also records an initialization sequence, which includes at least one ordered register write operation item, each of which records the register address, the data to be written, and the delay parameter; Performing input / output operations on the target peripheral includes: initializing the target peripheral according to the initialization sequence, and reading raw data from the target peripheral according to the target register address and the read data length; The method further includes: before returning the execution result to the large language model, the general-purpose device tool decodes the original data into physical quantity values ​​according to the decoding formula, and uses the physical quantity values ​​as the execution result.

5. The method according to claim 1, characterized in that, The general-purpose device tools include device reading tools, device writing tools, and device control tools; The device reading tool is used to read data from the target peripheral device; The device writing tool is used to write data or configuration parameters to the target peripheral device; The device control tool is used to control the target peripheral to perform actions according to the control sequence recorded in the device description file corresponding to the target peripheral.

6. The method according to claim 1, characterized in that, The method further includes: During the operation of the terminal device, the device description file corresponding to the newly added peripheral is loaded from the file system of the terminal device; The loaded device description file is parsed, and the device identifier recorded in the loaded device description file is registered as an available value of the device identifier in the calling parameters of the general device tool.

7. The method according to claim 2, characterized in that, The candidate set also includes at least one skill item, each skill item being defined by a skill description file, which records the triggering conditions, execution step sequence and parameter constraint rules, but does not record any peripheral hardware parameters; When the target candidate is a skill item, the parameter mode description corresponding to the target candidate includes the parameter mode description of the general equipment and tools involved in the execution step sequence; and The method further includes: sequentially receiving and executing tool calls to the general device tool initiated by the large language model according to the execution step sequence.

8. A tool recall system for an end-side device, characterized in that, Deployed on an end-side device connected to multiple peripherals, the system includes a device configuration layer, a general device tool layer, and a driver execution layer; The device configuration layer stores a device description file for each of the plurality of peripherals. The device description file records the hardware parameters of the corresponding peripheral in a structured format. The hardware parameters include interface protocol type, device address, and operation definition. The general device tool layer registers general device tools, which use device identifiers and operation identifiers as invocation parameters. Multiple peripherals share these general device tools. The general device tool layer receives tool invocations initiated by the large language model in response to user requests. These tool invocations carry the device identifier of the target peripheral and the operation identifier of the operation to be performed. The general device tool layer also searches the device configuration layer for the target peripheral based on the device identifier carried in the tool invocation. It parses the interface protocol type, device address, and operation definition corresponding to the operation identifier carried in the tool invocation from the found device description file, and then invokes the driver corresponding to the interface protocol type in the driver execution layer. The driver execution layer includes drivers categorized by interface protocol type. The invoked driver performs input / output operations on the target peripheral according to the device address and the operation definition, and returns the execution result of the input / output operations to the general device tool layer. The general-purpose device tool layer is also used to return the execution result to the large language model.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images