Method and system for realizing AI processing capability of operating system

Abstract

The invention discloses a method and system for realizing the AI processing capability of an operating system, and the method comprises the steps: carrying out the variable definition of a captured user instruction through an AI adapter, and transmitting a variable to an intention engine in a logic processing engine; an intention engine in the logic processing engine carries out corresponding processing through a cue word engine in the logic processing engine to generate a reasoning stream, and the reasoning stream is sent to the AI model for processing; when the AI model processes and executes the reasoning flow, calling the corresponding AI model for reasoning; and generating a task flow according to a reasoning result, wherein each execution process in the task flow is executed by calling a corresponding task execution module and tool through the AI adapter. According to the invention, the AI capability is added to the OS of the traditional terminal, and a multi-device platform and a multi-operating system are supported; various AI models such as language models, visual models, audio models, database indexes and the like are supported; the plug-in flexible adaptation of the AI model is supported; and the data privacy problem is avoided as far as possible by taking an end-side model as a main

Description

Technical Field

[0001] The present invention relates to the field of operating system technology, and in particular to a method and system for implementing AI processing capabilities of an operating system, a computing device, and a storable medium. Background Art

[0002] With the rapid development of artificial intelligence (AI), especially the development of large AI models, the operating systems (OS) of traditional mobile phones, car computers and other embedded devices, with hardware resource management and application scheduling as their core functions, can no longer meet the new demands of the AI ​​era for system intelligence, service personalization and natural interaction.

[0003] The few technologies currently available for adding AI capabilities to operating systems (OSs) are either primarily used on computers or only enhance local AI functionality on a single platform. These technologies have limited scope and are primarily applied to relatively single platforms or devices, or they only offer single LLM (Large Language Model) capabilities. Furthermore, these technologies lack flexibility in adapting AI models. Because current technologies are often strongly coupled with large AI models, they lack the flexibility to update AI models and capabilities. Furthermore, existing technologies primarily rely on large cloud-based models, which pose data privacy issues and lead to a lack of data security. Summary of the Invention

[0004] One of the purposes of the embodiments of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a method and system, computing device and storable medium for realizing the AI ​​processing capabilities of an operating system. For mobile phones, car computers and other embedded devices under multiple systems such as Linux, Android, QNX, etc., a method for adding AI capabilities to the operating system OS is provided. By integrating AI model capabilities, the OS is given more intelligent, more efficient and more natural interactive capabilities.

[0005] To solve the above technical problems, in a first aspect, an embodiment of the present invention provides a method for implementing AI processing capabilities of an operating system, the method comprising: The captured user instructions are defined as variables through the AI ​​adapter, and the variables are sent to the intent engine in the logic processing engine; The intention engine of the logic processing engine performs corresponding processing through the prompt word engine in the logic processing engine to generate an inference flow, and sends the inference flow to the AI ​​model for processing; When the AI ​​model processes and executes the inference flow, the corresponding AI model is called to perform inference; A task flow is generated based on the inference results, and each execution process in the task flow is executed by calling the corresponding task execution module through the AI ​​adapter.

[0006] Preferably, when the AI ​​model processes and executes the reasoning flow, calling the corresponding AI model for reasoning specifically includes: Calling a package plug-in for AI model processing, wherein the package plug-in includes a session management interface and a model inference interface; Creating a model session through the session management, and executing reasoning tasks on the input reasoning stream through the model reasoning interface; The prompt word engine transfers the inference results to the logic processing engine.

[0007] Preferably, performing the reasoning task on the input reasoning flow through the model reasoning interface specifically includes: Through the prompt word engine interface of the corresponding logic processing engine, the inference flow is used as the input for AI model processing; The AI ​​model processes and executes model reasoning, completes reasoning through the model reasoning interface, and generates commands; The generated command is returned to the intent engine in the logic processing engine through the prompt word engine.

[0008] Preferably, generating a task flow according to the inference result, and executing each execution process in the task flow by calling a corresponding task execution module through the AI ​​adapter specifically includes: After receiving the command, the intent engine of the logic processing engine determines whether the command is executable by comparing the current command with the pre-registered whitelist of supported functions. If so, the command is passed as an action flow string to the intelligent agent engine of the logic processing engine for multi-task orchestration. The multi-task orchestration generates a serial graph of execution nodes, each node is an execution process, and each execution process is executed by calling the corresponding module through the AI ​​adapter.

[0009] Preferably, each execution process is executed by calling corresponding modules and tools through the AI ​​adapter, specifically including: For tasks that require AI capabilities, the AI ​​adapter calls the agent task processing; The agent task processing passes the task prompt word to the prompt word engine, calls the corresponding model processed by the AI ​​model, and the model reasoning interface calls the required sensor engine, and the sensor engine calls the OS module to perform sensor data collection; The model reasoning interface combines the data collected by the sensor and the task prompt words of the prompt word engine to perform reasoning to obtain the reasoning result; Return the reasoning result to the agent engine in the logic processing engine and update the task execution workflow; The updated task execution workflow is passed to the OS module and then returned to the AI ​​adapter's agent task processing, calling the registered fixed task interface, and the task execution is completed. The AI ​​adapter's agent task processing returns a flag indicating that the execution is completed.

[0010] In a second aspect, to solve the technical problem of the present invention, an embodiment of the present invention further provides a system for implementing AI processing capabilities of an operating system, the system comprising: An OS module, an AI model processing module, a logic processing module, and an AI adapter, wherein the OS module is respectively connected to the AI ​​model processing module and the intelligent agent engine submodule in the logic processing module, the AI ​​model processing module is respectively connected to the sensor engine submodule in the logic processing module and the prompt word engine submodule in the logic processing module, the prompt word engine submodule in the logic processing module is respectively connected to the intention engine submodule in the logic processing module, and the AI ​​adapter is respectively connected to the sensor engine submodule in the logic processing module and the prompt word engine submodule in the logic processing module, and the intention engine submodule in the logic processing module; The AI ​​model processing module supports large AI models, including various model inference engines and is encapsulated into a model inference interface; The sensor engine submodule of the logic processing module is used to schedule and manage sensors; The prompt word engine submodule of the logic processing module is used to perform prompt word caching, prompt word splicing, prompt word triggering reasoning, and provide the results returned by the reasoning interface of the AI ​​model processing module; The intent engine submodule of the logic processing module is used to convert the input into formatted text commands, that is, to feed the input into the corresponding model plug-in and return the formatted text commands generated by the model; The agent engine submodule of the logic processing module is used to generate a serial diagram of execution nodes from the formatting command, where each node in the diagram is an execution process, and each execution process is executed by calling the corresponding module through the AI ​​adapter; The AI ​​adapter is used to perform tasks of the agent function.

[0011] Furthermore, the intention engine sub-module of the logic processing module is also used to compare the pre-registered support function whitelist based on the input formatted text command to determine whether the task is executable. If it is executable, the above formatted text command is passed as an action flow string to the intelligent agent engine of the logic processing module for multi-task orchestration.

[0012] In a third aspect, an embodiment of the present invention further provides a computing device comprising a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method as described above.

[0013] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor or a calculator, the processor executes the method as described above.

[0014] Compared with the existing technology, the system and method for realizing the AI ​​processing capability of an operating system provided by the embodiments of the present invention have at least the following beneficial effects: it solves the problem of a single application scope and can support application on multiple operating systems on multiple device platforms, such as Linux, Android, and QNX systems on devices from Qualcomm, MediaTek, NVIDIA, etc.; it solves the problem of a single AI capability and can support multiple types of AI model capabilities, such as LLM (Large Language Model), VLM (Vision-Language Model), ASR (Automatic Speech Recognition), TTS (Text-to-Speech), RAG (Retrieval-Augmented Generation), etc.; it solves the problem of AI model adaptation flexibility. When applying large AI models, the technical solution of the embodiments of the present invention can adopt a plug-in design to support flexible use or replacement of models; it solves the data privacy issue of cloud models. The technical solution of the embodiments of the present invention mainly uses local offline lightweight terminal-side models, which largely avoids the data privacy issue of cloud models. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The preferred embodiments will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of the present invention.

[0016] Figure 1 This is a system block diagram of an embodiment of the present invention for implementing the AI ​​processing capability of an operating system; Figure 2 This is a flow chart of a method for implementing AI processing capabilities of an operating system according to an embodiment of the present invention; Figure 3 The figure is a schematic diagram of the structure of a computing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings and other embodiments can be obtained based on these drawings without inventive work.

[0018] To simplify the drawings, only the parts related to the invention are schematically shown in each figure. They do not represent the actual structure of the product. In addition, to simplify the drawings and facilitate understanding, in some figures, only one of the components with the same structure or function is schematically depicted or labeled. In the embodiments of the present invention, "one" not only means "only one" but also "more than one". The following mainly uses some specific embodiments as examples to describe in detail the implementation of the technical solution of the present invention.

[0019] like Figure 1 As shown, in order to solve the technical problem of the present invention, an embodiment of the present invention further provides a system for realizing AI processing capabilities of an operating system, the system comprising: An OS module, an AI model processing module, a logic processing module, and an AI adapter, wherein the OS module is respectively connected to the AI ​​model processing module and the intelligent agent engine submodule in the logic processing module, the AI ​​model processing module is respectively connected to the sensor engine submodule in the logic processing module and the prompt word engine submodule in the logic processing module, the prompt word engine submodule in the logic processing module is respectively connected to the intention engine submodule in the logic processing module, and the AI ​​adapter is respectively connected to the sensor engine submodule in the logic processing module and the prompt word engine submodule in the logic processing module, and the intention engine submodule in the logic processing module; The AI ​​model processing module supports large AI models, including various model inference engines and is encapsulated into a model inference interface; The sensor engine submodule of the logic processing module is used to schedule and manage sensors; The prompt word engine submodule of the logic processing module is used to perform prompt word caching, prompt word splicing, prompt word triggering reasoning, and provide the results returned by the reasoning interface of the AI ​​model processing module; The intent engine submodule of the logic processing module is used to convert the input into formatted text commands, that is, to feed the input into the corresponding model plug-in and return the formatted text commands generated by the model; The agent engine submodule of the logic processing module is used to generate a serial diagram of execution nodes from the formatting command, where each node in the diagram is an execution process, and each execution process is executed by calling the corresponding module through the AI ​​adapter; The AI ​​adapter is used to perform tasks of the agent function.

[0020] Furthermore, the intention engine sub-module of the logic processing module is also used to compare the pre-registered support function whitelist based on the input formatted text command to determine whether the task is executable. If it is executable, the above formatted text command is passed as an action flow string to the intelligent agent engine of the logic processing module for multi-task orchestration.

[0021] The following embodiments are used to introduce the main technical solutions of the present invention, based on the appended Figure 1 The following explanation is given in a bottom-up manner, that is, in the order of OS module, AI model processing module, logic processing module, and AI adapter, to explain in detail how to implement the above modules.

[0022] For OS modules: Reuse the corresponding hardware devices, such as Qualcomm, MediaTek, other MCU / SOC, OS systems such as Linux, Android, QNX, and then provide sensor call C++ interface, mainly through C++ open source libraries such as OpenCV vision library, SDL2 multimedia library, etc., to realize the call of sensor driver, send hardware sensor call request to OS, and realize the call of sensor hardware.

[0023] For the AI ​​model processing module: (1) The AI ​​model processing module supports large AI models (such as LLM, VLM, etc.) and model inference engines (such as llama.cpp, whisper.cpp, etc.), and rewrites the inference engine and encapsulates it into a new inference interface.

[0024] For example, based on the llama.cpp inference engine, new C++ inference interfaces are encapsulated, such as the following main interfaces: platform_model_plugin::Create(EngineConfig config) is used to read the configuration file to prepare to create the inference environment; platform_model_plugin::Inference(InferenceData& data, Error** err, std::string &result) is used to call the interface implemented in llama.cpp by passing inference parameters to perform model inference and return the inference result; platform_model_plugin::InferenceAsync(InferenceData& data) also calls the interface implemented in llama.cpp to perform model inference, and the result is returned in real time using a callback function.

[0025] The underlying calls to sensor C++ open source libraries (such as OpenCV and SDL2), calls to related processor backends (such as CPU / NPU / GPU), and calls to related system libraries (such as libc++.so and libm.so) are all implemented in the inference engine llama.cpp and are directly called when calling model inference.

[0026] (2) Then, an abstraction layer is implemented through C++, which can call different models using the same function interface to implement functions such as calling model reasoning, plug-in management, and session management of model reasoning.

[0027] For example, for the main interface of plugin management: createPlugin() creates a new plug-in object; releasePlugin() deletes the plugin object.

[0028] Main interfaces for model inference: Abstract Create, Inference, InferenceAsync, etc. directly call the corresponding plug-in interface in the platform_model_plugin in the plug-in.

[0029] Main interface for session management: createSession(AiModelInfo& modelInfo, int sessionId, std::stringsessionName) creates a thread to call the model inference interface by reading the configuration, session ID and name; releaseSession(int sessionId) releases the inference thread by session id.

[0030] Encapsulate the above content into a model plug-in so, such as lib_whisper_plugin.so.

[0031] For the sensor engine submodule of the logic processing module Based on open-source sensor C++ libraries (such as OpenCV, SDL2, and ASLA), C++ function interfaces are written to schedule and manage sensors. These interfaces implement functions such as sensor device identification and selection, sensor opening, closing, and pausing, and sensor data input and output. When the sensor engine is called, a hardware sensor call request is sent to the OS based on the incoming task type (such as language, vision, or audio) to request the required sensor resources.

[0032] For the prompt word engine submodule of the logic processing module Through C++ code, we mainly implement the function interface of prompt word caching, prompt word splicing, prompt word triggering reasoning, and obtaining reasoning return.

[0033] For example, CachePrompt(const std::string& prompt) appends the user or system prompt words to the internal cache std::vector; BuildPrompt() uses C++ string method to assemble several prompt words into a complete prompt word in sequence; TriggerInference((InferenceData& data, Error** err,) gets the prompt and input data as the data parameter, and then calls the inference interface in the model plug-in for inference; GetInferenceResult() returns the inference result through a callback function, or retrieves the result from the internal buffer and returns it.

[0034] For the intent engine submodule of the logic processing module Through C++ code, it mainly implements input conversion to formatted text commands, reasoning calls, and rationality evaluation function interfaces. For example: The InputToCommand(const InputData& raw_input) API converts input data to a formatted text command. It determines whether the input is a simple input stream (such as a text stream or an audio stream) or a text command in a specified format. If the input is a simple input stream, a prompt word is passed to invoke a fixed inference process to convert the input into a text command in a specified format (such as "[{id:1,type:ai,tool:vlm_model,task: scene_mood}, {…}, …]"). For example, a text stream invokes LLM, an audio stream invokes ASR+LLM, a video stream invokes VLM+LLM, and so on.

[0035] Inference call interface: InvokeModel(const std::string& prompt, ModelType model) is called by InputToCommand. Based on the result of judging the input type, it feeds the prompt word / audio feature / video frame to the corresponding model plug-in and returns the formatted text command generated by the model.

[0036] Rationality assessment interface: ValidateCommand(const std::string& json_cmd, std::string& registry_cmd) compares the input formatted text command with the pre-registered support function whitelist registry_cmd to evaluate whether the task is executable. If executable, the above formatted command is passed as an action flow string to the intelligent agent engine for multi-task orchestration.

[0037] For the agent engine submodule of the logic processing module Use Taskflow to draw a serial diagram of execution nodes for the formatted command; each node is an execution process for calling external tools, and each execution process is executed by calling the corresponding module and tool through the AI ​​adapter.

[0038] Through C++ code, we mainly implement the LLM reasoning task graph, generate the execution task flow, and execute the graph function interface.

[0039] (1) LLM reasoning task graph: InferTaskGraph(const std::string& json_cmd, const std::vector <toolmeta>& tools) sends the user's original formatting commands and system prompts (including a list of available tools and constraints) to the LLM, which then directly returns a task graph describing the task execution graph (nodes, edges, conditions, and parallel relationships). This corresponds to a simple process with fewer than three steps, where the API directly calls the generated execution task flow.

[0040] (2) Generate execution task flow: parse the task graph returned by LLM, create tf::Task for each node using the taskflow library, and establish dependencies or branches according to the edges; (3) Execution Graph: RunGraph(tf::Taskflow& g, Context& ctx) passes the Taskflow generated in the previous step to tf::Executor for execution; when the node is executed, it dynamically calls the external tool through the AI ​​adapter and writes the result back to the Context.

[0041] For AI Adapter (1) AI Client SDK is responsible for traditional non-agent functions. It is equivalent to directly storing the function registry and directly calling OS functions without AI. The main function interface implementation is as follows: RegisterNormalTask(const std::string& tool, const std::string& task,std::function<int(const Json::Value&, std::string&)> func) Register the type:normal task function pointer into the global table. When calling, directly use the tool name and task name to look up the table to get the std::function and execute it; ExecuteNormal(const Json::Value& cmd, std::string& result) When type==normal in the formatted text command, the AI ​​adapter directly calls this interface, looks up the internal table, calls the specific function, and writes the result back to result.

[0042] (2) The AI ​​Agentic SDK is responsible for tasks that require agent functionality. It first dlopens the model plugin, then calls the sensor engine to obtain data, feeds the prompt words and data into the model for inference, and finally updates the workflow. The main function interface implementations are as follows: LoadPlugin(const std::string& plugin_path) originally dlopen corresponds to so.

[0043] CaptureSensor(const std::string& sensor_type, const std::string&task_hint) triggers the corresponding sensor (camera / microphone, etc.) according to the task description and returns the sensor data or file path; Infer(void* plugin_handle, const std::string& prompt, const Json::Value& data) sends the prompt word and data to the loaded model plug-in and returns the model inference result; UpdateWorkflow(const std::string& new_cmd_json) generates the next one or more commands based on the inference results and updates them directly to the workflow queue of the agent engine.

[0044] like Figure 2 To achieve the purpose of the present invention, an embodiment of the present invention provides a method for realizing AI processing capabilities of an operating system, the method comprising: S1. Define variables of captured user instructions through the AI ​​adapter and send the variables to the intent engine in the logic processing engine; S2. The logic processing intention engine performs corresponding processing through the prompt word engine in the logic processing engine to generate an inference flow, and sends the inference flow to the AI ​​model processing module; S3. When the AI ​​model processes and executes the inference flow, the corresponding AI model is called for inference; S4. Generate a task flow based on the inference result. Each execution process in the task flow is executed by calling the corresponding task execution module through the AI ​​adapter.

[0045] Preferably, when the AI ​​model processes and executes the reasoning flow, calling the corresponding AI model for reasoning specifically includes: Calling a package plug-in for AI model processing, wherein the package plug-in includes a session management interface and a model inference interface; Creating a model session through the session management, and executing reasoning tasks on the input reasoning stream through the model reasoning interface; The prompt word engine transfers the inference results to the logic processing engine.

[0046] Preferably, performing the reasoning task on the input reasoning flow through the model reasoning interface specifically includes: Through the prompt word engine interface of the corresponding logic processing engine, the inference flow is used as the input for AI model processing; The AI ​​model processes and executes model reasoning, completes reasoning through the model reasoning interface, and generates commands; The generated command is returned to the intent engine in the logic processing engine through the prompt word engine.

[0047] Preferably, generating a task flow according to the inference result, and executing each execution process in the task flow by calling a corresponding task execution module through the AI ​​adapter specifically includes: After receiving the command, the intent engine of the logic processing engine determines whether the command is executable by comparing the current command with the pre-registered whitelist of supported functions. If so, the command is passed as an action flow string to the intelligent agent engine of the logic processing engine for multi-task orchestration. The multi-task orchestration generates a serial graph of execution nodes, each node is an execution process, and each execution process is executed by calling the corresponding module through the AI ​​adapter.

[0048] Preferably, each execution process is executed by calling corresponding modules and tools through the AI ​​adapter, specifically including: For tasks that require AI capabilities, the AI ​​adapter calls the agent task processing; The agent task processing passes the task prompt word to the prompt word engine, calls the corresponding model processed by the AI ​​model, and the model reasoning interface calls the required sensor engine, and the sensor engine calls the OS module to perform sensor data collection; The model reasoning interface combines the data collected by the sensor and the task prompt words of the prompt word engine to perform reasoning to obtain the reasoning result; Return the reasoning result to the agent engine in the logic processing engine and update the task execution workflow; The updated task execution workflow is passed to the OS module and then returned to the AI ​​adapter's agent task processing, calling the registered fixed task interface, and the task execution is completed. The AI ​​adapter's agent task processing returns a flag indicating that the execution is completed.

[0049] Next, the present invention combines specific application scenarios to illustrate the mutual interaction process between the various modules of the present invention as much as possible. In fact, the technical solution of the present invention has a relatively wide range of applications, so this application scenario is only used as an example to illustrate the interactive relationship between the various modules of the technical solution of the present invention.

[0050] Scenario description: A user issues a voice command, "Please play music suitable for the current environment." The user hopes that by running a system equipped with this invention, the user can capture images from a camera, use the VLM model to infer the environment tag, and then search for and play music of the same tag type. The specific process is as follows: (1) Capturing voice commands: The system OS module calls the microphone software library, such as the ALSA (Advanced Linux Sound Architecture) audio API, to call the microphone hardware device to capture the voice input command "Please play appropriate music according to the current environment atmosphere", and passes its audio stream as input to the intent engine for processing in the form of audio variables defined in the AI ​​adapter.

[0051] (2) Voice command processing flow: The intent engine provides a C++ interface. If it determines that the task input is an independent audio type data, it will trigger the inference flow of the pre-solidified voice-to-text command, that is, through C++ code, the prompt word string of the inference flow of the ASR+LLM model and the acquired audio data are passed to the prompt word engine, and then the above data is passed to the AI ​​model processing module for inference to execute the command to convert the audio input into the specified format text.

[0052] (3) ASR speech-to-text: When the AI ​​model processing module executes the inference flow of the speech-to-text command, it receives the audio and prompt words (such as "Convert this audio stream to text") from the prompt word engine through C++ variables and calls an ASR model (such as the whisper model) for inference. Whisper reasoning is done by calling the model encapsulation plug-in lib_whisper_plugin.so through dlopen. lib_whisper_plugin.so is based on the API transformation of the inference engine such as whisper.cpp, and finally provides external session management interfaces (such as createSession, releaseSession, etc.), plug-in management interfaces (such as createPlugin, releasePlugin, etc.), model reasoning interfaces (such as Create, Inference, InferenceAsync, etc.) and other APIs; When using the whisper model, the API is called to create a model session and load a whisper model plug-in. Then, the inference interface is called to pass in the obtained prompt word and audio stream to perform the inference task, and finally the inference result text "Please play appropriate music according to the current environment atmosphere" is returned to the prompt word engine.

[0053] (4) LLM text to formatted command: The prompt word engine gets the text returned by ASR reasoning and passes it to the AI ​​model processing module as input to execute LLM, such as Qwen model reasoning. Through the prompt word engine C++ interface, the text data "Please play appropriate music according to the current environment atmosphere" and the prompt word "text to formatted text command" are passed to the Qwen model for reasoning; similar to the previous one, through the dlopen method, the C++ API in the model encapsulation plug-in lib_qwen_plugin.so is called to complete the reasoning and convert the text to formatted text command, such as "1. Take a photo and VLM recognizes the environment atmosphere (such as xx); 2. Search for xx type music and play it". The corresponding formatted text command is "[{id:1,type:ai,tool:vlm_model, task: scene_mood},{id:2,type:normal,tool:music_player,task:play_music_{{step1.scene_mood}}}]". Then the formatted text command is returned to the intent engine through the prompt word engine.

[0054] (4) Text command intention judgment: After the intent engine receives the formatted text command, it determines whether it is executable by comparing the current command task operation with the pre-registered support function whitelist. If it is executable, the above formatted command is passed as an action flow string to the intelligent agent engine for multi-task orchestration.

[0055] (5) Intelligent task orchestration: Use Taskflow to draw a serial diagram of execution nodes for the formatted command; each node is an execution process for calling external tools, and each execution process is executed by calling the corresponding modules and tools through the AI ​​adapter.

[0056] (6) Task execution: After the task list is arranged, the intelligent agent engine will pass it to the OS module as a workflow for execution. For tasks that do not require AI capabilities, the AI ​​Client SDK in the AI ​​adapter will be called for execution. The type value of the formatted command is used to determine whether the tasks that do not require AI capabilities are "type:normal". The AI ​​Client SDK in the AI ​​adapter will be called for execution. For tasks that require AI capabilities are "type:ai", the AI ​​Agentic SDK in the AI ​​adapter will be called for execution.

[0057] (7) Execute Task 1 - Take a photo and use VLM to identify the ambient atmosphere: The AI ​​adapter passes the task "{id:1,type:ai,tool:vlm_model,task: scene_mood}" to the AI ​​Agentic SDK for execution through the type value. The AI ​​Agentic SDK passes the task prompt word, i.e., the task field, to the prompt word engine, and then calls the VLM model of the AI ​​model processing module, such as the MobileVLM model. It also calls lib_mobilevlm_plugin.so in the dlopen manner. The model inference engine code calls the sensor engine through the OpenCV library, and finally calls the OS module to control the camera to take a photo and return the image data. The MobileVLM model takes the image and the prompt word "scene_mood" for inference, and finally outputs the current ambient atmosphere, such as the atmosphere of "relaxation". It then returns it to the intelligent agent engine and updates the task "2. Search for relaxing music and play it" formatted command to "{id:2,type:normal,tool:music_player,task: play_music_relax}".

[0058] (8) Execute Task 2 - Search for and play relaxing music: The updated task is passed to the OS module as an execution workflow and then returned to the AI ​​Client SDK of the AI ​​adapter. The registered fixed task interface is called, such as the C++ API of playMusic(relax). playMusic(relax) first searches for a playlist of the corresponding type in the specified folder, and then uses system() to call mpg123 to play the music in the relaxing playlist.

[0059] (9) Execution completed: After all tasks are completed, the AI ​​Client SDK of the AI ​​adapter returns a completion flag. This marks the end of the workflow execution.

[0060] This invention embeds a modular AI capability framework into the traditional operating system (OS), giving various terminal devices (mobile phones, car computers, embedded devices, etc.) intelligent task processing, natural interaction and multi-model collaboration capabilities, while ensuring data privacy and system flexibility. Specifically, it includes: 1. Cross-platform, multi-system AI empowerment: Unified integration of AI capabilities for multiple operating systems (Linux / Android / QNX, etc.) on different hardware platforms (Qualcomm / MediaTek / NVIDIA, etc.), breaking device and OS limitations.

[0061] 2. Multimodal AI Capability Integration and Invocation: Supports invoking and managing various AI models, including Large Language Models (LLMs) for complex language understanding and generation. Visual Language Models (VLMs) enable cross-modal understanding of images and text. ASR (ASR) converts speech to text. TTS (Text-to-Speech) converts text to natural speech. RAG (Retrieval-Augmented Generation) enhances answer accuracy by integrating external knowledge.

[0062] 3. Plug-in model management and flexible adaptation: A plug-in model design allows developers and users to flexibly access, replace, or upgrade different AI models. The AI ​​model processing module is responsible for creating model inference sessions and dynamically calling the corresponding model plug-in for inference based on input (prompt word).

[0063] 4. Device-first privacy-preserving inference: This approach primarily relies on local, offline lightweight models to perform inference tasks, significantly reducing cloud data transmission and protecting the privacy of user sensitive data (such as images, audio, and conversation content) from the source.

[0064] 5. Intelligent end-to-end task processing engine: Achieve intelligent task parsing, orchestration, and execution through logic processing modules.

[0065] 6. Unified developer interface (AI adapter SDK): Provides a standardized SDK function interface as a bridge between upper-level applications or services and the AI ​​framework of the present invention, simplifying the task submission and data transfer process.

[0066] In a third aspect, an embodiment of the present invention further provides a computing device, which includes a processor and a memory, the memory being used to store a computer program, the computer program including program instructions, and the processor or calculator being configured to call the program instructions to execute the method as described above.

[0067] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor or a calculator, the processor or the calculator executes the method as described above.

[0068] like Figure 3 As shown, an embodiment of the present application provides a computing device, which includes a processor or a calculator (not shown in the figure) 1001 and a memory 1002. The processor or calculator 1001 and the memory 1002 can be interconnected through a communication bus 1003. The communication bus 1003 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1003 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the memory 1002 is used to store a computer program, which includes program instructions. The processor 1001 is configured to call the program instructions, and the above program includes instructions for executing some or all of the steps in the aforementioned method.

[0069] The processor 1001 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above program.

[0070] The memory 1002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0071] The computing device 1000 may further include a communication module 1004 and a display 1005. The communication module 1004 may be connected to the optical tracking device for communication. The communication module 1004 may be a wireless communication module (eg, a WiFi module, a Bluetooth module, etc.) or a wired communication module.

[0072] In addition, the computing device 1000 may also include common components such as a communication interface (eg, a USB interface, a microphone interface, etc.), an antenna, etc., which will not be described in detail here.

[0073] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0074] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0076] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0077] In addition, the functional units in the various embodiments of the application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software program modules.

[0078] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0079] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which may include: a flash drive, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.

[0080] The above is a detailed introduction to the embodiments of the present application. Specific examples are used in the embodiments of the present invention to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0081] It should be noted that the above embodiments can be freely combined as needed. The above are only preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention.< / toolmeta>

Claims

1. A method for realizing AI processing capability of an operating system, characterized in that: The method comprises: The captured user instructions are defined as variables through the AI ​​adapter, and the variables are sent to the intent engine in the logic processing engine; The intention engine in the logic processing engine performs corresponding processing through the prompt word engine in the logic processing engine to generate an inference flow, and sends the inference flow to the AI ​​model for processing; When the AI ​​model processes and executes the inference flow, the corresponding AI model is called to perform inference; A task flow is generated based on the inference results, and each execution process in the task flow is executed by calling the corresponding task execution module through the AI ​​adapter.

2. The method for realizing AI processing capability of an operating system according to claim 1, wherein: When the AI ​​model processes and executes the inference flow, calling the corresponding AI model for inference specifically includes: Calling a package plug-in for AI model processing, wherein the package plug-in includes a session management interface and a model inference interface; Creating a model session through the session management, and executing reasoning tasks on the input reasoning stream through the model reasoning interface; The prompt word engine transfers the inference results to the logic processing engine.

3. The method for realizing AI processing capability of an operating system according to claim 2, wherein: The performing of the reasoning task on the input reasoning flow through the model reasoning interface specifically includes: Through the prompt word engine interface of the corresponding logic processing engine, the inference flow is used as the input for AI model processing; The AI ​​model processes and executes model reasoning, completes reasoning through the model reasoning interface, and generates commands; The generated command is returned to the intent engine in the logic processing engine through the prompt word engine.

4. The method for realizing AI processing capability of an operating system according to claim 3, wherein: The task flow is generated according to the inference result, and each execution process in the task flow is executed by calling the corresponding task execution module through the AI ​​adapter, specifically including: After receiving the command, the intent engine of the logic processing engine determines whether the command is executable by comparing the current command with the pre-registered whitelist of supported functions. If so, the command is passed as an action flow string to the intelligent agent engine of the logic processing engine for multi-task orchestration. The multi-task orchestration generates a serial graph of execution nodes, each node is an execution process, and each execution process is executed by calling the corresponding modules and tools through the AI ​​adapter.

5. The method for realizing AI processing capability of an operating system according to claim 4, wherein: Each execution process is executed by calling corresponding modules and tools through the AI ​​adapter, specifically including: For tasks that require AI capabilities, the AI ​​adapter calls the agent task processing; The agent task processing passes the task prompt word to the prompt word engine, calls the corresponding model processed by the AI ​​model, and the model reasoning interface calls the required sensor engine, and the sensor engine calls the OS module to perform sensor data collection; The model reasoning interface combines the data collected by the sensor and the task prompt words of the prompt word engine to perform reasoning to obtain the reasoning result; Return the reasoning result to the agent engine in the logic processing engine and update the task execution workflow; The updated task execution workflow is passed to the OS module and then returned to the AI ​​adapter's agent task processing, calling the registered fixed task interface, and the task execution is completed. The AI ​​adapter's agent task processing returns a flag indicating that the execution is completed.

6. A system for implementing AI processing capabilities of an operating system, characterized in that: The system comprises: An OS module, an AI model processing module, a logic processing module, and an AI adapter, wherein the OS module is respectively connected to the AI ​​model processing module and the intelligent agent engine submodule in the logic processing module, the AI ​​model processing module is respectively connected to the sensor engine submodule in the logic processing module and the prompt word engine submodule in the logic processing module, the prompt word engine submodule in the logic processing module is respectively connected to the intention engine submodule in the logic processing module, and the AI ​​adapter is respectively connected to the sensor engine submodule in the logic processing module and the prompt word engine submodule in the logic processing module, and the intention engine submodule in the logic processing module; The AI ​​model processing module supports large AI models, including various model inference engines and is encapsulated into a model inference interface; The sensor engine submodule of the logic processing module is used to schedule and manage sensors; The prompt word engine submodule of the logic processing module is used to perform prompt word caching, prompt word splicing, prompt word triggering reasoning, and provide the results returned by the reasoning interface of the AI ​​model processing module; The intent engine submodule of the logic processing module is used to convert the input into formatted text commands, that is, to feed the input into the corresponding model plug-in and return the formatted text commands generated by the model; The agent engine submodule of the logic processing module is used to generate a serial diagram of execution nodes from the formatted text command, where each node in the diagram is an execution process, and each execution process is executed by calling the corresponding module through the AI ​​adapter; The AI ​​adapter is used to perform tasks of the agent function.

7. The system for implementing the AI ​​processing capability of an operating system according to claim 6, wherein: The intention engine submodule of the logic processing module is also used to compare the pre-registered support function whitelist based on the input formatted text command to determine whether the task is executable. If it is executable, the above formatted text command is passed as an action flow string to the intelligent agent engine of the logic processing module for multi-task orchestration.

8. A computing device, characterized in that The computing device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor or the computer executes the method according to any one of claims 1 to 5.

Images