AI intelligent agent system and working method

Abstract

The invention provides an AI intelligent agent system, and the system comprises a perception processing module which is used for receiving external perception information and processing the external perception information through a corresponding model, and obtaining corresponding perception data; the cache middleware module is used for storing process data, and the process data comprises the perception data and / or context data; the agent scheduling module is used for acquiring a user instruction, acquiring the stored process data from the cache middleware module, filling the instruction and the process data into a cue word template, and delivering the cue word template to the LLM after fine adjustment; the LLM after fine tuning is used for reasoning according to the content of the cue word template to generate a reasoning result. The invention further provides a corresponding working method of the AI agent system. According to the method, the interpretability, the maintainability, the higher generalization ability and the higher response speed of modular design can be achieved at the same time.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence interaction technology, and in particular to an AI intelligent agent system and its working method. Background Technology

[0002] The improved semantic understanding capabilities of large language models make it possible for artificial intelligence to provide services to users in diverse scenarios. For example, AI models can provide embodied intelligent interactive experiences by perceiving the physical world. This usually requires a comprehensive model that can process multimodal information such as images and text simultaneously, such as the Visual Language Model (VLM) based on the Transformer architecture.

[0003] Existing technologies generally adopt two mainstream architectures: "end-to-end" and "hierarchical decision-making," both of which have obvious limitations in practical applications.

[0004] End-to-end architectures like VLM obtain text and visual feature vectors through text and visual encoders, and use a cross-attention mechanism to enable the model to understand information from images and text. While possessing strong generalization capabilities, they are essentially "black box models," resulting in poor interpretability and controllability. Furthermore, this type of architecture heavily relies on large-scale, high-quality labeled data, and the computational resources required for training and inference are expensive, demanding a high level of expertise from algorithm developers.

[0005] The "layered decision-making" architecture adopts a design paradigm that is more in line with traditional software engineering, dividing the system into independent functional modules such as perception, understanding, and decision-making. Although this architecture is more in line with the modular and integrated design requirements of traditional software engineering, the information transmission between its various modules often uses complex hard coding and makes decisions based on fixed fields, which limits the overall generalization ability of the system. In practical application scenarios, the cost of system iteration and optimization is high.

[0006] Therefore, in this context, how to provide an AI intelligent agent system that combines the interpretability and maintainability of modular design with high generalization ability and fast response speed is a technical problem that needs to be solved. Summary of the Invention

[0007] In view of the above-mentioned problems of the prior art, this application provides an AI intelligent agent system and a working method, which can provide an AI intelligent agent system that has the interpretability and maintainability of modular design, as well as high generalization ability and fast response speed.

[0008] To achieve the above objectives, the first aspect of this application provides an AI intelligent agent system, comprising: The perception processing module is used to receive external perception information and process it by the corresponding model to obtain the corresponding perception data; A caching middleware module is used to store process data, including the perception data and / or context data; The intelligent agent scheduling module is used to obtain user instructions, retrieve the stored process data from the cache middleware module, and fill the instructions and process data into the prompt word template and deliver it to the fine-tuned LLM; The fine-tuned LLM is used to perform inference based on the content of the prompt word template and generate inference results.

[0009] In summary, by modularizing the perception processing, caching middleware, agent scheduling, and LLM functions, the input-output relationships between each module are clarified. While maintaining a clear system structure and ease of maintenance, the collaborative processing of physical environment perception and natural language interaction is achieved.

[0010] By introducing a caching middleware as an intermediate storage unit for the perception processing module, agent scheduling module, and LLM, the continuous perception information of the perception processing module can be recorded to form a perception history. This helps the LLM better understand the objective physical world and make inference decisions that are more in line with the needs of the display user. The agent scheduling module, by reading on demand, avoids the response latency caused by waiting for the output of the perception processing module serially, thus improving the overall throughput of the system.

[0011] Because LLM (Large Language Model) receives perceptual and contextual data through a unified prompt word template, its generalization ability does not depend on the internal implementation of specific modules, thus enabling flexible adaptation to various types of data input. The overall response efficiency of the system mainly depends on the inference performance of the LLM itself. This allows the architecture to achieve a more natural and efficient human-computer interaction experience while maintaining the maintainability and low iteration cost brought by modular design. Through the cooperation between the LLM model and environmental perception, the inference capability of the AI ​​agent system is not limited by the selection method of traditional visual language models (VLM). Theoretically, the LLM model base can be switched as needed at a small cost, reducing the training cost and GPU resource consumption of the model, and increasing the upper limit of capability.

[0012] As one possible implementation of the first aspect, the perception processing module includes at least one of the following: The speech processing module is used to receive speech information and convert it into text data through the ASR model; The vision processing module receives visual information and converts it into structured data using a CV model.

[0013] As described above, the speech processing module enables the perception processing module to receive speech commands and convert them into text data that LLM can understand, thus supporting multimodal human-computer interaction methods involving text and speech. The data output by the perception processing module is unified into machine-readable, semantically clear structured data, providing reliable input for the subsequent multimodal inference module.

[0014] As one possible implementation of the first aspect, the voice processing module is also used to receive voice commands, convert them into text commands through the ASR model, and then provide them as user commands to the agent scheduling module.

[0015] As described above, by receiving voice commands, a multimodal human-computer interaction method combining text and voice is constructed to adapt to the interaction needs in different scenarios, thereby improving the ease of use and applicability of the AI ​​agent.

[0016] As one possible implementation of the first aspect, it also includes: a TTS output module for text-to-speech conversion; When the reasoning result is in text format, the agent scheduling module is also used to call the TTS output module so that the TTS output module can convert the text-format reasoning result into speech format for broadcasting.

[0017] In summary, by converting the output of LLM into speech, a multimodal human-computer interaction method is constructed, which improves the ease of use and applicability of AI agents.

[0018] As one possible implementation of the first aspect, it also includes: an execution mechanism for performing an action; When the reasoning result is a structured instruction, the agent scheduling module is also used to call the execution mechanism so that the execution mechanism responds to the structured instruction and performs the corresponding action.

[0019] In summary, by utilizing the inference results of LLM and calling corresponding execution mechanisms such as "action capabilities," "internet search engine calls," and "online analysis capabilities" through the agent scheduling module, diversified functional expansion is achieved, thereby improving the usability and applicability of AI agents.

[0020] As one possible implementation of the first aspect, it also includes: a memory abstraction module, which performs semantic-level compression processing on the process data stored in the cache middleware to generate context data that retains key temporal states, event semantics and user intent.

[0021] As described above, the memory abstraction module performs semantic-level compression on the stored process data, which efficiently preserves the information of the process data and saves storage space.

[0022] As one possible implementation of the first aspect, it also includes: a data GC module, which performs deletion operations on process data stored in the cache middleware based on its expiration or redundant status.

[0023] Therefore, by regularly or quantitatively cleaning the stored process data, the storage function can be ensured to operate well in order to support high-speed read and write requirements.

[0024] As one possible implementation of the first aspect, it also includes: an adaptive datasparsification mechanism, which is used to perform sparsification processing on the process data stored in the cache middleware according to the time span from the current time.

[0025] As described above, by optimizing the storage of historical memory information for visual perception through the data sparsity module, such as saving recent data at high resolution and saving distant data at low resolution, the performance degradation caused by large data volume is avoided, and the ability to perceive recent key visual data and capture changes in visual information over time is maintained.

[0026] The second aspect of this application provides a method for operating an AI intelligent agent system, including the following steps: The perception processing module receives external perception information, which is then processed by the corresponding model to obtain the corresponding perception data. Process data is stored through a caching middleware module, and the process data includes the perception data and / or context data. The system obtains user instructions through the intelligent agent scheduling module, retrieves stored process data from the cache middleware module, and fills the instructions and process data into the prompt word template before delivering it to the fine-tuned LLM. The finely tuned LLM performs inference based on the content of the prompt word template to generate inference results.

[0027] A third aspect of this application provides a computing device, including a processor and a memory storing program instructions thereon, which, when executed by the processor, cause the processor to perform the operating method of the AI ​​intelligent agent system described in the second aspect.

[0028] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a computer, cause the computer to perform the working method of the AI ​​intelligent agent system described in the second aspect.

[0029] The fifth aspect of this application provides a computer program product including program instructions that, when executed by a computer, cause the computer to perform the working method of the AI ​​intelligent agent system described in the second aspect. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the AI ​​agent system provided in the first embodiment of this application; Figure 2a This is a flowchart of the working method of the AI ​​intelligent agent system provided in the second embodiment of this application; Figure 2b This is a system schematic diagram of the AI ​​agent provided in the second embodiment of this application; Figure 2c This is a schematic diagram of an AI agent including a language processing subsystem provided in the second embodiment of this application; Figure 2d This is a schematic diagram of the LLM fine-tuning method provided in the second embodiment of this application; Figure 3 This is a schematic structural diagram of a computing device provided in an embodiment of this application; Figure 4 This is an illustration of a prompt template after the content has been filled in.

[0031] It should be understood that the dimensions and shapes of the blocks in the above structural diagrams are for reference only and should not constitute an exclusive interpretation of the embodiments of the present invention. The relative positions and inclusion relationships between the blocks presented in the structural diagrams are only schematic representations of the structural relationships between the blocks, and are not intended to limit the physical connection methods of the embodiments of the present invention. Detailed Implementation

[0032] The technical solutions provided in this application will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the system architecture and business scenarios provided in the embodiments of this application are mainly for illustrating possible implementations of the technical solutions of this application and should not be construed as the sole limitation on the technical solutions of this application. Those skilled in the art will recognize that the technical solutions provided in this application are equally applicable to similar technical problems as system architectures evolve and new business scenarios emerge.

[0033] It should be understood that the AI ​​agent system solutions provided in the embodiments of this application include AI agent systems, methods for operating AI agent systems, computing devices, computer-readable storage media, and computer program products. Since these technical solutions solve problems based on the same or similar principles, some repetitions may not be repeated in the following descriptions of specific embodiments, but these specific embodiments should be considered as mutually referencing each other and can be combined with each other.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. In case of any inconsistency, the meaning set forth in this specification or derived from the content described herein shall prevail. Furthermore, the terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application. To accurately describe the technical content of this application and to accurately understand the invention, the following explanations or definitions of the terms used in this specification are provided before describing specific embodiments: 1) Large Language Model (LLM): A deep learning model trained on massive amounts of text data, typically using the Transformer architecture, capable of understanding, generating, summarizing, and reasoning about natural language text. Its "large" aspect is reflected in its enormous number of parameters (often in the billions to trillions), giving it powerful language understanding and context learning capabilities.

[0035] 2) Vision-Language Model (VLM): A type of multimodal model capable of simultaneously processing and understanding visual (e.g., images, videos) and linguistic (e.g., text) information. It achieves image-based question answering, description, reasoning, and other tasks by aligning or fusing visual and linguistic features in a shared semantic space.

[0036] 3) Low-Rank Adaptation (LoRA): A typical technique for efficient parameter fine-tuning. It assumes that the changes in model weights during task adaptation have low-rank characteristics. Therefore, it simulates weight updates by injecting the product of two trainable low-rank matrices into the original weights, thereby significantly reducing the number of trainable parameters.

[0037] 4) Mixed Precision Training: A deep learning training optimization technique that combines floating-point numbers with different numerical precisions during training. Typically, lower precision (e.g., FP16 / BF16) is used for most computations and storage to improve speed and reduce memory usage, while higher precision (e.g., FP32) is used to maintain key variables (such as weight updates) to ensure numerical stability and final accuracy during training.

[0038] 5) YOLO (You Only Look Once): A real-time visual object detection algorithm. Its core idea is to construct the object detection task as a single regression problem, using a neural network to directly predict multiple bounding boxes and their categories from a complete image in a single forward propagation. This series of algorithms is known for its good balance between speed and accuracy.

[0039] 6) Automatic Speech Recognition (ASR): A technology that automatically converts human speech signals into corresponding text content. It analyzes the input audio waveform using acoustic and language models to identify words and sentences contained in the speech. Streaming ASR enables simultaneous listening and transcription, reducing interaction latency.

[0040] 7) Text-to-Speech (TTS): A technology that converts written text into understandable artificial speech. It generates speech waveforms that conform to language rhythm and specific timbre through steps such as text analysis and acoustic model synthesis. Streaming TTS can generate and play back simultaneously, improving the real-time performance and fluency of voice interaction.

[0041] 8) Voice Activity Detection (VAD): A technique for automatically identifying and distinguishing speech segments from non-speech segments (such as silence and background noise) in continuous audio signals. VAD modules are typically deployed at the front end of an ASR system, analyzing the energy, spectrum, and other characteristics of the audio to determine whether the current frame contains valid human voices. Its core function is to filter irrelevant noise, reduce unnecessary subsequent processing overhead, and provide precise start and end points for streaming interaction, thereby improving the efficiency, real-time performance, and robustness of the voice interaction system.

[0042] 9) Dynamic Post-Training Quantization: This refers to a compression technique that dynamically determines quantization parameters (such as scaling factors) based on runtime data after the model has been trained, without retraining or fine-tuning. Compared to static quantization, which requires a fixed calibration set, dynamic quantization adapts better to changes in input data. Typically, activation values ​​are dynamically quantized, while weights can be quantized statically or dynamically.

[0043] 10) HQQ (Half-Quadratic Quantization): A post-training weight quantization method that does not require calibration data. This method constructs a robust optimization objective based on semi-quadratic loss to directly compress the pre-trained model weights at low bit depth (e.g., 4-bit). Its main feature is that the quantization process does not rely on an additional representative calibration dataset, thus simplifying the workflow, and it has faster processing speed and better model compatibility.

[0044] 11) GPTQ (GPT Quantization): A high-precision post-training quantization method based on second-order information. This method typically requires layer-by-layer quantization of the model and utilizes a small calibration dataset to minimize quantization error in order to find the optimal quantization parameters for the weights of each layer. It performs well in maintaining the accuracy of the model after low-bit (e.g., 4-bit) quantization.

[0045] 12) AWQ (Activation-aware Weight Quantization): An activation-aware weight quantization method. Its core idea is to protect the more important weights (usually associated with higher activation amplitudes) that have a greater impact on the model output, using higher quantization precision for these weights, while using more aggressive quantization for less important weights. This method also requires a small calibration dataset to evaluate the activation distribution.

[0046] 13) Tool: In this application, the model itself cannot be called, but it can be called and a data interface is provided after being encapsulated as a module. The Tool of a certain model can be understood as a module that encapsulates a certain model, has the functions of that model, and can be scheduled by the AI ​​Agent scheduling module (i.e., AI Agent Core). For example, CV Tool is a module that encapsulates a CV model.

[0047] 14) Prompt Template: In this application, a structured text-based prompt template guides the fine-tuned LLM to correctly understand, process, and respond to data from multimodal modules (such as CV and ASR), while constraining its identity, behavior, and output format. Specifically, the prompt template can be a structured text skeleton containing several placeholders or slots, which can be used to inject formatted CV data, ASR data, dialogue memory, human commands, and other information. Figure 4 A prompt template is shown, with its placeholders or slots already filled with content.

[0048] The AI ​​agent system solution provided in this application includes: a perception processing module for receiving external perception information and processing it by a corresponding model to obtain corresponding perception data; a cache middleware module for storing process data, including perception data and / or context data; and an agent scheduling module for acquiring user instructions, obtaining the stored process data from the cache middleware module, and filling the instructions and process data into a prompt word template before delivering it to a fine-tuned LLM. The fine-tuned LLM is used to perform inference based on the content of the prompt word template to generate inference results. This system combines the interpretability and maintainability of modular design with high generalization ability and fast response speed. This application embodiment can be applied to human-computer interaction systems in various fields such as human-computer interaction, multimodal robots, and intelligent control centers. The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0049] The first embodiment of this application provides an AI intelligent agent system, which will be described below in conjunction with... Figure 1 The implementation methods of each component of the system are described in detail, including: The perception processing module is used to receive external perception information and process it by the corresponding model to obtain the corresponding perception data.

[0050] A caching middleware module is used to store process data, including the perception data and / or context data; The intelligent agent scheduling module is used to obtain user instructions, retrieve the stored process data from the cache middleware module, and fill the instructions and process data into the prompt word template and deliver it to the fine-tuned LLM; The fine-tuned LLM is used to perform inference based on the content of the prompt word template and generate inference results.

[0051] In some embodiments, the sensing input module includes at least one of the following: The speech processing module is used to receive speech information and convert it into text data through the ASR model; The vision processing module receives visual information and converts it into structured data using a CV model.

[0052] In some embodiments, visual information about the environment is received through visual acquisition hardware. Depending on the application scenario, the visual acquisition hardware may be a camera or video camera equipped with different lenses such as wide-angle lenses, fisheye lenses, and HDR lenses. The visual acquisition hardware may be hardware on robotic equipment such as robots or drones, or hardware mounted in a fixed position with a third-person perspective. In industrial inspection scenarios, line scan cameras or 3D structured light cameras may also be integrated to obtain higher precision depth or three-dimensional information.

[0053] In some embodiments, the acquired visual information may be in formats such as HDR photos, frames extracted from videos, or point cloud data.

[0054] In some embodiments, visual information is processed using a visual model. The visual model can be a pre-trained lightweight object detection model such as the YOLO series (e.g., YOLO-v6, YOLO-v7, YOLO-v8), an anchor-free or Transformer-based lightweight model, etc. Specifically, the lightweight object detection model can be fine-tuned using a specific image dataset to enhance its ability to recognize target objects in the target scene. The specific image dataset can be manually labeled using tools such as LabelStudio.

[0055] In some embodiments, the structured data obtained by transforming visual information includes at least one of the following: frame information, bounding box coordinate information, item category information, and sampling time information.

[0056] In some embodiments, visual information is cleaned by writing the cleaned visual information in full to a cache, or writing it to a cache at preset sampling intervals. Data cleaning includes at least one of the following operations: data standardization and formatting, filtering out invalid data, adding timestamps and frame association, data compression, data simplification and enhancement, etc.

[0057] In some embodiments, the voice processing module is further configured to receive voice commands, convert them into text commands through an ASR model, and provide them as user commands to the agent scheduling module.

[0058] In some embodiments, the time spent processing silence and noise is reduced by integrating ASR technology with voice activity detection (VAD) technology.

[0059] In some embodiments, it further includes: a TTS output module for implementing text-to-speech; When the reasoning result is in text format, the agent scheduling module is also used to call the TTS output module so that the TTS output module can convert the text-format reasoning result into speech format for broadcasting.

[0060] In some embodiments, it further includes: an actuator for performing an action; When the reasoning result is a structured instruction, the agent scheduling module is also used to call the execution mechanism so that the execution mechanism responds to the structured instruction and performs the corresponding action.

[0061] In some embodiments, the caching middleware can be implemented by a high-performance, low-latency storage system for temporarily storing visual information and context. Specific implementations may include in-memory key-value databases (such as Redis, Memcached), message streaming platforms (such as Apache Kafka), or in-memory data grids (such as Apache Ignite).

[0062] In some embodiments, it further includes: a memory abstraction module, used to perform semantic-level compression processing on the process data stored in the cache middleware to generate context data that retains key timing states, event semantics and user intent.

[0063] The contextual data is stored in text form according to a custom time strategy, such as summarizing the first 50 rounds of dialogue between the user and the AI ​​agent.

[0064] In some embodiments, the multimodal reasoning module invokes the corresponding dialogue memory or visual information for reasoning only when the user mentions content related to dialogue memory or visual information.

[0065] In some embodiments, the system further includes a data garbling module (Data GC) for performing deletion operations on process data stored in the cache middleware based on its expired or redundant status.

[0066] In some embodiments, the system further includes an adaptive data sparsification mechanism for sparsifying the process data stored in the caching middleware based on the time span from the current time.

[0067] In some embodiments, visual information can be divided into multiple stages according to time, and different sampling intervals can be used to retain each stage; preferably, the sampling interval is longer for stages that are further away from the current time.

[0068] In some embodiments, the prompt template combines the user instruction, visual information, and contextual data in a preset format as input to the LLM. For example, the prompt template could be: "You are an AI agent. The current visual scene is as follows: [Visual Information]. Our previous conversation history summary is: [Conversation Memory]. Now, the user says: [User Input]. Please respond based on the above information." In some embodiments, the LLM is fine-tuned through the following steps: Obtain a pre-trained LLM; Construct a training dataset containing the target role's identity information; The pre-trained LLM is trained on the training dataset using a parameter-efficient fine-tuning technique. Among them, the parameter efficient fine-tuning technology limits the number of trainable model parameters to a preset threshold; the training adopts a mixed precision training strategy, using a first numerical precision to store the model parameters, and performing forward and backward propagation calculations, and using a second numerical precision not lower than the first numerical precision to update the model parameters.

[0069] In some embodiments, a supervised fine-tuning dataset is constructed, which includes the target role identity, specific task instructions, and expected output format. The fine-tuning dataset may include various user queries that the AI ​​agent needs to respond to and their corresponding visual context descriptions.

[0070] In some embodiments, efficient parameter fine-tuning techniques can be implemented using the Low-Rank Adaptation (LoRA) method. This method achieves efficient task adaptation by injecting trainable low-rank matrix pairs (denoted as A and B) into specific linear layers in the original LLM (such as feedforward networks, query / key / value projections in self-attention modules), while keeping the original model parameters frozen.

[0071] In some embodiments, mixed-precision training can be achieved by using a combination of half-precision floating-point numbers and single-precision floating-point numbers. Specifically, the first numerical precision can be half-precision floating-point numbers (such as FP16 or BF16 format) used to store model parameters and perform forward and backward propagation calculations; the second numerical precision can be single-precision floating-point numbers (FP32) used to accumulate gradients, perform weight updates, and store some core variables that are sensitive to numerical precision.

[0072] In some embodiments, after the LLM completes the efficient fine-tuning of the parameters, it is further compressed using a post-training quantization method to reduce the storage and computing resource requirements when deploying the model.

[0073] In some embodiments, the post-training quantization method is a dynamic quantization method that does not require calibration data. Preferably, it employs a semi-quadratic quantization (HQQ) method that supports 1-8 bit quantization. This method directly performs low-bit compression (e.g., compression to 4-bit) on the model weights through fast and robust optimization, offering faster processing speed and broader model architecture compatibility compared to methods that require calibration datasets (such as GPTQ and AWQ).

[0074] The second embodiment of this application provides a method for operating an AI intelligent agent system. The following will refer to... Figure 2a The flowchart shown illustrates that the method provided in this second embodiment includes the following steps S200-S230.

[0075] S200: Continuously collects environmental visual information, processes it through a fine-tuned visual model, stores the output data in a cache middleware, and retains it through a data sparsity mechanism to obtain visual information.

[0076] The robotic device continuously acquires video data streams of the surrounding physical environment via cameras. To achieve real-time target detection of the video stream, the system typically includes a video frame extraction and scheduling module. This module is responsible for extracting image frames from the raw video stream according to a preset strategy and inputting these image frames one by one into a fine-tuned visual model.

[0077] This application takes YOLO-v8 as an example and makes minor adjustments to it to give it a stronger ability to recognize target items in the target scene.

[0078] First, construct a custom image dataset using an image annotation tool (such as LabelStudio). For the target scene, acquire or collect images containing the objects to be identified, and use the annotation tool to accurately annotate the bounding boxes of these objects and their corresponding category labels in the images.

[0079] Using the aforementioned labeled custom dataset, the pre-trained YOLO-v8 model was fine-tuned. The training process was conducted on a training set containing target scene data, and the model parameters were adjusted by optimizing the loss function to enable it to learn and enhance its ability to recognize the visual features of target objects.

[0080] After training, the performance metrics (such as mean average precision, mAP) of the fine-tuned model are evaluated on an independent validation set. Once the performance meets the requirements, the final model weight file is deployed to the perception processing module, and the fine-tuned model is used to perform object detection on the input image frame data to obtain visual information.

[0081] For example, a YOLO-v8 model performs object detection on an image frame (e.g., frame 100) and identifies a car and a billboard. After enabling YOLO-v8's tracking function (using model.track and setting persist=True) to confirm the recognition correlation of the above objects across multiple frames, the output data is as follows:

[0082] The image frame information, sampling time, bounding box information, and item category information used to output the above data are clearly grouped into a set of structured data objects that can be understood and parsed by LLM. The cleaned output data will be fully written to the cache middleware (such as...). Figure 2b (As shown in the Cache Data Pool), serving as the full visual memory of the LLM.

[0083] Caching middleware is a caching module that enables data exchange and parallel read / write between the visual model and the LLM (Local Management Model). To achieve fast response times, storage services such as Redis can be used as caching middleware, enabling rapid read / write operations through their provided APIs. On one hand, it eliminates the need for the LLM's input to wait sequentially for the visual model's output; on the other hand, it provides a foundation for the LLM to select high-quality visual information and build memory.

[0084] To prevent the storage space of the cache middleware from being exhausted and to improve the confidence of visual information, embodiments of this application also provide a data sparsity mechanism.

[0085] Based on practical experience, full visual memory can clog the context window of an LLM (Local Mode Module), causing AI agents to perform poorly in long, multi-turn dialogues. In fact, distant visual memories do not require excessively fine granularity; therefore, data sparsity is used to clean the full visual memory. For example, within a specified past time span, only visual memories within 5 seconds of the current time are fully retained. For visual memories between 5 and 120 seconds, a coarser granular retention method is used, such as retaining data every 5 frames. For visual memories between 120 seconds and 10 minutes, a larger interval retention method is used, such as every 30 frames. Data exceeding this time span is directly deleted. Understandably, the above parameters are configurable and can be adjusted by those skilled in the art according to actual conditions.

[0086] Data that has undergone sparsification is visual information.

[0087] S210: Receive user input and construct prompt information based on the user input.

[0088] Users can interact with the AI ​​agent system via text or voice. For example... Figure 2c As shown, the AI ​​intelligent agent system integrates a speech processing subsystem to support voice interaction functions. This speech processing subsystem includes an automatic speech recognition module for converting the user's real-time speech stream into text, which can employ ASR technology based on an end-to-end architecture or a traditional hybrid model. By integrating Voice Activity Detection (VAD) technology, the time spent processing silence and noise is reduced. The ASR technology uses streaming transmission, sending the converted text segments to the LLM (Local Level Manager) while simultaneously using asynchronous data storage to send the text segments to a caching middleware. That is, during user voice input, the text segments recognized by ASR are sent to the LLM and the caching middleware in parallel, eliminating the need for prior storage before processing by the LLM, resulting in higher efficiency.

[0089] The user-input (and converted) text information, the visual information stored in the cached middleware, and the context data (dialogue memory) stored in the previous dialogue will be used to construct prompt information through prompt word templates. Among them, [self-awareness] is the self-awareness assigned by LLM training, and the specific training method is detailed in step S220.

[0090] For example, the prompt template could be: "You are a [self-awareness]. The current visual scene is as follows: [visual information]. Our previous conversation history summary is: [conversation memory]. Now, the user says: [user input]. Please respond based on the above information." The context data is generated from multi-turn dialogue records between the AI ​​agent and the user. For the AI ​​agent, a complete record of all interactions with the user is usually not necessary. In certain scenarios, due to the context window limitations of the local LLM, a complete record of interactions can actually encroach on the token space used by the LLM for critical inference. Therefore, this application also introduces a long dialogue summarization tool, namely a memory abstraction module, for semantic-level compression of the context data.

[0091] The long conversation summarization tool summarizes the multiple rounds of dialogue between the AI ​​agent and the user, creating concise summaries. These summaries are stored in text format according to a custom time strategy; for example, the first 50 rounds of dialogue between the user and the AI ​​agent, totaling 1000 words, can be summarized into a single text. The summarized text might be only one sentence long, such as: "Previously, I discussed the development direction of embodied intelligence (VLA) with the user, and the user believes that VLA may not be the optimal solution for embodied intelligence."

[0092] Because visual information is organized and represented in a structured format (such as data objects containing fields like object category, coordinates, and confidence level), this type of data can be converted into natural language descriptions or normalized text that can be parsed by LLM using more complex cue word templates. For example: "The current visual scene contains the following objects: {object_list}. Among them: {object_1} is located in the frame at {position_1} with a confidence level of {confidence_1}. {object_2} is located in the frame at {position_2} with a confidence level of {confidence_2}." Wherein, object_list is a list of identified objects, object_1 and object_2 are identified object 1 and object 2 respectively, position_1 and position_2 are the coordinates or positions of object 1 and object 2 respectively, and confidence_1 and confidence_2 are the confidence scores of identified object 1 and object 2.

[0093] When the AI ​​agent receives user input, it simultaneously calls the cache middleware read / write API and LLM to organize visual information, dialogue memory, and user input with prompt word templates into prompt information and delivers it to the LLM for inference.

[0094] By constructing the aforementioned prompts, the LLM can carry visual information and dialogue memory acquired in real time by the visual model during each inference. However, inference based on visual information and dialogue memory can also be triggered by user interaction. For example, the LLM will only retrieve the relevant visual information and dialogue memory for inference when the user explicitly asks, "Do you remember what we talked about three minutes ago?" Otherwise, the LLM will only retrieve the visual information from the most recent frames and the data from the most recent rounds of dialogue for combined inference.

[0095] S220: The fine-tuned LLM performs inference based on the prompt information and generates inference results.

[0096] The prompt information constructed in step S210 is input into the fine-tuned LLM, so that the LLM can perform inference based on the prompt information and generate the inference result.

[0097] The following is a brief introduction to the fine-tuning method of LLM, including steps S221-S225.

[0098] S221: Obtain the pre-trained LLM and construct the training dataset.

[0099] This application uses Qwen / Qwen2.5-7B-Instruct as the base for pre-trained LLM. The SWIFT framework is used as the fundamental tool for fine-tuning and quantization.

[0100] The training dataset is implemented by integrating multiple open-source datasets (such as alpaca-gpt4-data-zh, alpaca-gpt4-data-en, and self-cognition).

[0101] The training set aims to give the model a clear sense of self, including the model name, the trainer's identity / organization, and the tasks it excels at (such as handling interactive tasks combined with visual context, or recognizing rectangular objects such as squares of various colors and books). This allows for high customization of the LLM model.

[0102] S222: Training is performed using LoRA fine-tuning technology and a mixed precision strategy.

[0103] This embodiment employs a low-rank adaptive (LoRA) parameter fine-tuning technique combined with a hybrid precision strategy for training.

[0104] LoRA technology involves injecting trainable low-rank matrix pairs into all linear layers of the model (such as feedforward networks and query / key / value projection layers in self-attention modules) while keeping the original model weights frozen. The rank of the low-rank matrix is ​​set to 8, and the scaling factor is 32. This configuration strictly limits the number of trainable parameters to a preset threshold well below the total number of model parameters (typically less than 1%), achieving efficient task adaptation.

[0105] During training, a BF16 mixed-precision training strategy is employed. Specifically, BF16 half-precision floating-point format is used to store model parameters and activation values, and most of the computations in forward and backward propagation are performed. Simultaneously, single-precision floating-point numbers (FP32) are used internally by the optimizer to accumulate gradients, calculate weight updates, and maintain a copy of the master weights in FP32 precision. This strategy significantly reduces GPU memory usage and improves computational throughput with almost no loss of training accuracy.

[0106] The following shows a portion of the training scripts.

[0107]

[0108] S223: Perform training quantization on the fine-tuned model.

[0109] As shown in part of the training script in step S222, this embodiment of the application uses the semi-quadratic quantization method (HQQ) to perform post-training quantization compression on the model.

[0110] HQQ, a dynamic quantization method supporting flexible bit width configurations from 1 to 8 bits, utilizes its fast and robust optimization algorithm to directly and dynamically quantize model weights from the original FP16 precision to a specified low bit precision (e.g., 4-bit in this example) without relying on any additional calibration datasets. Compared to quantization methods that require calibration data (such as GPTQ and AWQ), HQQ offers faster processing speeds and stronger model architecture compatibility. After this 4-bit quantization, the memory usage of the model weights can theoretically be reduced to 25% of the original FP16 model.

[0111] This quantization step, combined with the BF16 mixed-precision training strategy used in step S223 above, creates a complementary effect in reducing memory requirements, and together achieves a significant optimization of the model's total memory usage during training and deployment.

[0112] Through the training steps described above, a fine-tuned LLM is obtained and deployed in the AI ​​agent system.

[0113] The AI ​​agent as a whole still possesses the same reasoning ability as the base model (such as being able to answer questions about professional knowledge such as "loss function" and "activation function" fluently), while also improving its ability to visually perceive the physical environment.

[0114] S230: Output the inference results to the user.

[0115] like Figure 2c As shown, the inference results can be converted from text to speech using TTS technology for streaming output, further reducing user waiting time. It can also be broadcast using a custom voice sample tone.

[0116] Figure 2b , 2c These are schematic diagrams of AI agents with and without language processing subsystems, respectively. The following are examples... Figure 2c The structure of the entire system will be explained using an example.

[0117] As shown in the figure, the AI ​​intelligent agent system (Self-Loop System) of this application embodiment includes: Perception processing module (ASR, CV); The AI ​​Agent Core module includes an LLM inference module fine-tuned by LoRA. Cache middleware module (including Cache Data Pool and its internal components); Output Feedback Module (TTS); Each module uses a cache data pool to achieve asynchronous data storage and read / write.

[0118] The following sections explain each module and the data relationships between them.

[0119] 1) Perception processing module, including: Voice processing module (ASR Tool): Receives external voice input, converts voice commands into text-formatted user commands using an automatic speech recognition (ASR) model, and writes the text to a cached data pool; The vision processing module (CV Tool) receives images / video streams captured by the camera, performs target detection and recognition using a computer vision model (such as YOLO-v8), outputs structured detection results (including category, location, confidence level, timestamp, etc.), and continuously stores them in the cache data pool in real-time.

[0120] 2) Caching middleware module, including: Cache Data Pool: Serving as the central data hub of the system, it is used to uniformly store and manage the following three types of process data: externally perceived information (from ASR, CV); context memory data (generated by the Memory Abstraction Tool); and system running status information (such as timestamps, task IDs, etc.). Data GC (Data Recycling Module): Monitors data timeliness and performs deletion operations on expired or redundant raw data to prevent memory overflow; Adaptive Data Sparsification Mechanism: Based on the time span of the data from the current moment, it dynamically adjusts the granularity of data retention, achieving adaptive compression of high-density data into low-dimensional semantic summaries.

[0121] The caching middleware module is used by all perception processing modules (ASR, CV) to write data to it; the AI ​​Agent Core module reads the required context from it; and the output feedback module indirectly depends on its output through TTS.

[0122] The Data Garbage Collection (Data GC) module is used to automatically clean up process data in the cached data pool. Specifically, it includes: monitoring whether data exceeds a preset time window (e.g., 5 seconds); checking whether the corresponding data has been digested by the memory abstraction module; and if the conditions are met, triggering a deletion operation to release storage space, thus preventing system performance degradation due to the accumulation of raw data. It works in conjunction with the memory abstraction module to implement a secure memory evolution mechanism of "digest first, then delete".

[0123] The Adaptive Data Sparsification Mechanism dynamically determines the retention strategy based on the time span of data from the current moment: for recent data (e.g., 0-5 seconds), it retains the full original data to support high-precision real-time inference; for older data (>5 seconds), it triggers sparsification processing, retaining only the high-level semantic summary. The degree of sparsification can be adaptively adjusted based on event saliency, user concerns, etc. This mechanism works in conjunction with Data GC to achieve efficient utilization of cached resources.

[0124] 3) The AI ​​Agent Core, located at the center of the system, is the control and decision-making hub of the entire Self-Loop System. It contains: LoRA-tuned LLM: It has the ability to understand multimodal contexts, perform semantic reasoning, and generate natural language; The Prompt-Template module provides a preset prompt word structure to integrate user commands, historical memory, and current sensory data. The Memory Abstraction Tool is responsible for retrieving raw data from the cache, calling the LLM to generate a semantic summary, and writing it back to the cache.

[0125] The agent scheduling module is responsible for receiving user instruction text from the ASR; reading relevant process data (such as recent visual records and dialogue summaries) from the cache data pool according to the instruction content; filling the above information into the prompt-Template to construct a complete context prompt; submitting it to the fine-tuned LLM for inference; obtaining the inference results (text format) output by the LLM; and calling the TTS module to play the speech if voice output is required.

[0126] 4) The Memory Abstraction Tool is used to perform semantic-level compression on high-density raw perceptual data (such as continuous frame detection results and multi-turn dialogue logs) stored in the cached data pool, and generate natural language summaries that retain key temporal states, event semantics and user intent.

[0127] Specifically, data to be processed is read from the cache data pool periodically or triggered by events; a dedicated prompt is built and submitted to the LLM; the LLM outputs a concise and readable summary text; and the summary is written back to the "sparse memory area" of the cache data pool to replace the original data as a long-term context representation.

[0128] This summary is used as context in subsequent LLM inference to ensure that the system has the ability to understand references across time (such as "Is it still there?").

[0129] 5) The TTS output module converts the text-formatted inference results generated by LLM into speech signals, which are then played through a speaker to complete the human-computer interaction loop. When the LLM output is text, the agent scheduling module automatically calls the TTS module to perform speech synthesis, achieving real-time speech feedback.

[0130] The process of using the above-mentioned AI agent system architecture to perform tasks includes: When a user issues a voice command, the ASR module captures and transcribes it into text, which is then written to the CacheData Pool.

[0131] The camera captures images, the CV module detects objects and outputs structured data, which is then written to the cache data pool in real time.

[0132] The Memory Abstraction Tool periodically scans the cache, generates a summary of timed-out data, and writes it to the cache. The Data GC module determines whether the original data can be deleted and performs cleanup. The Adaptive Data Sparsification Mechanism determines the granularity of data retention based on the time span.

[0133] When the AI ​​Agent Core receives a new instruction, it reads the latest visual data, dialogue summaries, etc. from the Cache DataPool; Then fill in the prompt template, construct the prompt, and submit it to the LLM; LLM generates natural language responses by combining context.

[0134] The AI ​​Agent Core calls the TTS module to return the information needed by the user via voice broadcast.

[0135] Figure 3 This is a schematic structural diagram of a computing device 900 provided in an embodiment of this application. This computing device can execute various optional embodiments of the methods described above. The computing device can be a terminal, or a chip or chip system within the terminal. Figure 3 As shown, the computing device 900 includes: a processor 910, a memory 920, and a communication interface 930.

[0136] It should be understood that Figure 3 The communication interface 930 in the computing device 900 shown can be used to communicate with other devices, and may specifically include one or more transceiver circuits or interface circuits.

[0137] The processor 910 can be connected to the memory 920. The memory 920 can be used to store the program code and data. Therefore, the memory 920 can be a storage unit inside the processor 910, an external storage unit independent of the processor 910, or a component that includes both the storage unit inside the processor 910 and the external storage unit independent of the processor 910.

[0138] Optionally, the computing device 900 may also include a bus. The memory 920 and communication interface 930 can be connected to the processor 910 via the bus. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The symbol is represented by a line without an arrow, but this does not mean that there is only one bus or one type of bus.

[0139] It should be understood that in the embodiments of this application, the processor 910 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. Alternatively, the processor 910 may employ one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0140] The memory 920 may include read-only memory and random access memory, and provides instructions and data to the processor 910. A portion of the processor 910 may also include non-volatile random access memory. For example, the processor 910 may also store device type information.

[0141] When the computing device 900 is running, the processor 910 executes computer execution instructions stored in the memory 920 to perform any of the operational steps of the above method and any of the optional embodiments thereof.

[0142] It should be understood that the computing device 900 according to the embodiments of this application can correspond to the corresponding subject in executing the methods according to the various embodiments of this application, and the above and other operations and / or functions of each module in the computing device 900 are respectively for implementing the corresponding processes of the methods of this embodiment. For the sake of brevity, they will not be described in detail here.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0148] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to perform the above-described method, which includes at least one of the schemes described in the above embodiments.

[0150] The computer storage medium in this application embodiment can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0152] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0153] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] Furthermore, the terms "first, second, third, etc." or similar terms such as module A, module B, and module C used in the specification and claims are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that, where permissible, a specific order or sequence may be interchanged so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0155] In the above description, the labels of the steps involved, such as S110, S120, etc., do not mean that the steps will necessarily be executed. The order of the steps can be interchanged or executed simultaneously if permitted.

[0156] The term "comprising" as used in the specification and claims should not be construed as limiting itself to what follows; it does not exclude other elements or steps. Therefore, it should be interpreted as specifying the presence of the mentioned feature, integral, step, or component, but does not exclude the presence or addition of one or more other features, integrals, steps, or components, or groups thereof. Thus, the statement "device comprising means A and B" should not be limited to a device consisting solely of components A and B.

[0157] The terms "an embodiment" or "an embodiment" as used in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in at least one embodiment of this application. Therefore, the terms "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily refer to the same embodiment, but may refer to the same embodiment. Furthermore, in one or more embodiments, the particular features, structures, or characteristics can be combined in any suitable manner, as will be apparent to those skilled in the art from this disclosure.

[0158] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, all of which fall within the scope of protection of this application.

Claims

1. An AI intelligent agent system, characterized in that, include: The perception processing module is used to receive external perception information and process it by the corresponding model to obtain the corresponding perception data; A caching middleware module is used to store process data, including the perception data and / or context data; The intelligent agent scheduling module is used to obtain user instructions, retrieve the stored process data from the cache middleware module, and fill the instructions and process data into the prompt word template and deliver it to the fine-tuned LLM; The fine-tuned LLM is used to perform inference based on the content of the prompt word template and generate inference results.

2. The system according to claim 1, characterized in that, The sensing processing module includes at least one of the following: The speech processing module is used to receive speech information and convert it into text data through the ASR model; The vision processing module receives visual information and converts it into structured data using a CV model.

3. The system according to claim 1, characterized in that, The voice processing module is also used to receive voice commands, convert them into text commands through the ASR model, and then provide them to the agent scheduling module as user commands.

4. The system according to claim 1, characterized in that, Also includes: The TTS output module is used to convert text to speech. When the reasoning result is in text format, the agent scheduling module is also used to call the TTS output module so that the TTS output module can convert the text-format reasoning result into speech format for broadcasting.

5. The system according to claim 1, characterized in that, Also includes: An executing mechanism is used to perform a certain action. When the reasoning result is a structured instruction, the agent scheduling module is also used to call the execution mechanism so that the execution mechanism responds to the structured instruction and performs the corresponding action.

6. The system according to claim 1, characterized in that, Also includes: The memory abstraction module is used to perform semantic-level compression processing on the process data stored in the cache middleware, generating context data that retains key temporal states, event semantics, and user intent.

7. The system according to claim 1, characterized in that, Also includes: The data recycling module is used to delete process data stored in the cache middleware based on its expiration or redundant status.

8. The system according to claim 1, characterized in that, Also includes: The data sparsification module is used to perform sparsification processing on the process data stored in the cache middleware based on the time span from the current time.

9. A method for operating an AI intelligent agent system, characterized in that, Includes the following steps: The perception processing module receives external perception information, which is then processed by the corresponding model to obtain the corresponding perception data. Process data is stored through a caching middleware module, and the process data includes the perception data and / or context data. The system obtains user instructions through the intelligent agent scheduling module, retrieves stored process data from the cache middleware module, and fills the instructions and process data into the prompt word template before delivering it to the fine-tuned LLM. The finely tuned LLM performs inference based on the content of the prompt word template to generate inference results.

10. A computing device, characterized in that, include: processor, and A memory storing program instructions that, when executed by the processor, cause the processor to perform the working method of the AI ​​intelligent agent system according to claim 8.

Images