Task execution method and device based on multi-modal model and electronic device

Abstract

The specification provides a remote sensing task processing method and device based on a multi-modal model and electronic equipment, which is applied to an agent system containing a multi-modal model and an image adapter; the system constructs a three-layer architecture of a tool execution layer, an intelligent decision layer and a system workflow, aiming to realize full automation of remote sensing interpretation through natural language instructions. The multi-modal model is fine-tuned as an intelligent hub, and its task planning and tool calling ability is trained through a special instruction data set; the multi-modal remote sensing image adapter is used to standardize the image and extract its metadata; the system receives user instructions, understands the intention by the multi-modal model, autonomously generates task planning, and drives the task engine to dynamically schedule and execute the bottom layer professional tools, through iterative execution of tool calling cycles to complete complex analysis, and finally returns the visual results and analysis report to the user. The application changes remote sensing analysis from professional programming to intelligent dialogue, significantly reduces the use threshold and improves the complex task solving ability.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the fields of artificial intelligence and remote sensing technology, and in particular to a task execution method, apparatus and electronic device based on a multimodal model. Background Technology

[0002] Remote sensing imagery carries rich information about the Earth's surface and is a key data source for major applications such as environmental monitoring and land planning. Automatically extracting information from imagery using intelligent interpretation technology is of great value. However, current technologies still have many shortcomings in achieving efficient and automated interpretation of complex remote sensing data.

[0003] Currently, deep learning-based remote sensing interpretation methods mainly suffer from the following problems: First, they provide dedicated algorithm tools with fixed functions and fragmented structures, requiring users to possess professional knowledge to manually combine and invoke them. This process is cumbersome and lacks automation, making it difficult to handle complex multi-step tasks. Second, directly applying general-purpose visual models for remote sensing image analysis has a significant domain gap, limiting its ability to understand non-visible light images such as synthetic aperture radar, and severely lacking the ability to invoke specialized tools, perform multi-step logical reasoning, and execute complex analysis tasks.

[0004] To address these issues, some advanced methods have emerged that utilize large language models or general multimodal models to generate tool call sequences based on a predefined tool list. However, these methods typically employ a static, pre-planning strategy, generating a complete and fixed execution plan at the start of the task. This planning model, lacking feedback and adjustment capabilities, appears rigid and fragile when faced with complex remote sensing tasks that require dynamic adjustments to subsequent steps based on intermediate execution results. It exhibits low fault tolerance and poor adaptability; if unexpected results occur in intermediate steps, the entire task chain may fail, leading to high trial-and-error costs and overall low efficiency.

[0005] Therefore, existing technologies are still unable to realize a one-stop intelligent analysis system that is centered on natural language interaction and capable of autonomous cognition, dynamic planning, and robust decision-making on multimodal remote sensing data. Summary of the Invention

[0006] In view of this, the purpose of this specification is to solve the technical problems of low automation, reliance on expert knowledge, low fault tolerance, poor adaptability, and inability to effectively handle complex dynamic tasks caused by the static planning strategy of existing remote sensing image analysis technology.

[0007] To achieve the above objectives, one or more embodiments of this specification provide a task processing method based on a multimodal model, applied to an intelligent agent system, the intelligent agent system including a multimodal model and an image adapter; the intelligent agent system has preset attribute descriptions for one or more tools; the method includes:

[0008] In response to a task command for an image input by the user, the image adapter is used to normalize the image and extract its metadata information to form image description text;

[0009] Based on the normalized image, the image description text, the attribute descriptions of one or more tools, and the task instructions, the multimodal model is invoked to perform inference, thereby generating a task plan for the task instructions and repeatedly executing the following steps until the task is completed or a preset loop limit is reached:

[0010] Step A: Based on the current context of the multimodal model, generate the next operation instruction using the multimodal model based on the task planning;

[0011] Step B: Parse the next operation instruction. If the next operation instruction is a call instruction for the target tool, then call the target tool and update the execution result of the target tool to the current context, and then return to step A; if the next operation instruction is a task completion instruction, then obtain the final execution result summarized and output by the multimodal model in the task completion instruction.

[0012] The final execution results are then integrated and output to the user terminal.

[0013] More preferably, the invocation parameters of the one or more tools are preset in the system prompts of the multimodal model; or, the invocation parameters of the one or more tools are stored in a tool knowledge base connected to the multimodal model.

[0014] More preferably, during the execution of the method, the current execution state of the intelligent agent system is output to the user terminal so as to visualize the current execution state on the user terminal.

[0015] More preferably, the data structure of the current context maintains the role-based dialogue sequence when the multimodal model performs the task planning;

[0016] The invocation instruction of the target tool is encapsulated in the dialogue sequence and in the content corresponding to the role of the multimodal model, and the execution result of the target tool is encapsulated in the dialogue sequence and in the content corresponding to the role of the tool library.

[0017] More preferably, the data structure of the current context maintains the world state when the multimodal model executes the task planning; the world state includes the identifier of the task instruction, the current execution result of the task planning, and the remaining execution target of the task planning.

[0018] More preferably, the multimodal model is obtained based on the following supervised fine-tuning process:

[0019] Construct a supervised fine-tuning dataset for the target domain; wherein, the training samples in the supervised fine-tuning dataset include the labels of training images, the description of training tasks, the attribute descriptions of several tools, and the role-based dialogue sequence; the dialogue sequence maintains the execution plan of the training task corresponding to the model role, and several rounds of dialogue sequence set according to the execution order of the training task, including the tool calling instructions corresponding to the model role and the execution results of the target tool corresponding to the tool role;

[0020] The training images are processed using the image adapter according to a preset standardization method.

[0021] Based on the training images processed according to the preset specifications and the supervised fine-tuning dataset, the pre-trained multimodal model is subjected to supervised fine-tuning to obtain the multimodal model.

[0022] More preferably, the construction of the supervised fine-tuning dataset in the remote sensing field includes:

[0023] Construct a tool knowledge base for the target domain, wherein the tool knowledge base contains meta-information of the tools;

[0024] Using a generative model, a description of the training task, attribute descriptions of the tools, and the role-based dialogue history are generated in the training samples based on the meta-information of the tools and the type of the training task.

[0025] The training samples are validated by experts to construct the supervised fine-tuning dataset.

[0026] More preferably, the step of using a generative model to generate a description of the training task, attribute descriptions of the tools, and the role-based dialogue history in the training samples based on the meta-information of the tools and the type of the training task includes:

[0027] Using the generative model, different training samples are generated for the same training task based on the meta-information of the tools. The different training samples differ in any one or more aspects of the description of the training task, the selection of the set of tools, their attribute descriptions, and the role-based dialogue history.

[0028] One or more embodiments in this specification also provide a task processing device based on a multimodal model, applied to an intelligent agent system, the intelligent agent system including a multimodal model and an image adapter; the intelligent agent system has preset attribute descriptions for one or more tools; the device includes:

[0029] An image adaptation unit is used to respond to a task command for an image input by a user terminal, to perform normalization processing on the image using the image adapter, and to extract its metadata information to form image description text.

[0030] The task execution unit is configured to, based on the normalized image, the image description text, the attribute descriptions of one or more tools, and the task instructions, invoke the multimodal model to perform inference, generate a task plan for the task instructions, and repeatedly execute the following steps until the task is completed or a preset loop limit is reached:

[0031] Step A: Based on the current context of the multimodal model, generate the next operation instruction using the multimodal model based on the task planning;

[0032] Step B: Parse the next operation instruction. If the next operation instruction is a call instruction for the target tool, then call the target tool and update the execution result of the target tool to the current context, and then return to step A; if the next operation instruction is a task completion instruction, then obtain the final execution result summarized and output by the multimodal model in the task completion instruction.

[0033] The output unit is used to integrate the final execution result and output it to the user terminal.

[0034] One or more embodiments of this specification provide an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor executes the executable instructions to perform the steps as described in any of the above embodiments of the strategy model optimization method based on parameter fine-tuning.

[0035] The methods, apparatuses, and computer devices described in one or more embodiments of this specification, through a dynamic iterative task planning and tool scheduling mechanism, can autonomously handle complex multi-step tasks, overcoming the low fault tolerance of static planning methods, significantly enhancing robustness, and realizing the ability to solve complex problems. By fine-tuning image adapters and domain instructions, the cognitive capabilities of general-purpose large models are organically integrated with the execution capabilities of specialized remote sensing tools, solving the problem of difficulty in understanding specialized images. Remote sensing analysis is transformed from complex expert programming into intuitive natural language dialogue, greatly reducing the technical threshold. Multiple tools for multimodal model invocation can be built in a microservice format, allowing for independent addition, deletion, and upgrades, enabling continuous evolution of system capabilities. Attached Figure Description

[0036] Figure 1 This is an exemplary embodiment of a general flowchart of a remote sensing task execution method based on a multimodal model.

[0037] Figure 2 This is a flowchart of a method for constructing a supervised fine-tuning dataset for a specific remote sensing domain, provided by an exemplary embodiment.

[0038] Figure 3 This is a flowchart of a method for training a multimodal model provided in an exemplary embodiment.

[0039] Figure 4 This is a flowchart of an exemplary embodiment of a method for operating an intelligent agent system based on a multimodal model.

[0040] Figure 5 This is a schematic diagram of a training sample data structure provided in an exemplary embodiment.

[0041] Figure 6 This is a flowchart of a method for executing a tool loop in an intelligent agent system based on a multimodal model, provided in an exemplary embodiment.

[0042] Figure 7 This is a schematic diagram of the structure of an image adapter provided in an exemplary embodiment.

[0043] Figure 8 This is a schematic diagram of an intelligent agent system structure provided in an exemplary embodiment.

[0044] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0046] In one embodiment provided in this specification, an intelligent agent system developed for task execution in the field of remote sensing is implemented to perform remote sensing tasks based on a multimodal model. The multimodal model described in this specification typically refers to an upgraded version of a large language model with at least millions of parameters, exhibiting intelligent emergent characteristics, breaking the limitations of a single modality, and capable of processing instructions in multiple formats such as natural language, images, audio, and video. The intelligent agents described in the various embodiments of this specification perceive changes in the environment, reason and make decisions based on their learned knowledge and algorithms, and then execute actions to influence the environment or achieve predetermined goals. Figure 8 As shown, the intelligent agent system (hereinafter sometimes referred to as the system) may include the following core modules: A perception module: used to acquire environmental information, such as receiving natural language commands input by the user and multimodal remote sensing images; A reasoning and decision-making module (intelligent hub): as the core of the system, this embodiment uses a multimodal model trained with data specific to the remote sensing field, responsible for understanding the task, planning steps, and generating tool invocation decisions; An execution module: a task engine used to schedule and execute tools and implement the task execution method described in this embodiment. The above system can be manifested as a computer device (local or cloud) with corresponding control code deployed, or it can be distributed and deployed in different functional modules; this specification does not limit this.

[0047] The solution provided in this embodiment constructs a dedicated supervised fine-tuning dataset and performs domain-adaptive training on a multimodal model, thereby deploying a dynamically iterative intelligent agent system to automate complex remote sensing analysis tasks. Those skilled in the art will understand that the large-scale model training methods, intelligent agent system deployment, and inference processes provided in this specification are not only applicable to the remote sensing field but can also be extended to other professional fields involving image processing tasks. Therefore, the remote sensing fields described in the following embodiments should not constitute a limitation on the technical solutions provided in this specification.

[0048] Reference Figure 1This document presents a flowchart illustrating the overall process of a remote sensing task execution method based on a multimodal model, as provided in the embodiments of this specification. Specifically, the method includes: step S100 of constructing a supervised fine-tuning dataset in the remote sensing domain; step S200 of training a multimodal model capable of invoking remote sensing tools; and step S300 of deploying and running the aforementioned intelligent agent system based on the multimodal model. These three steps together constitute a complete technical closed loop from data preparation and model empowerment to final application services.

[0049] First, step S100 is executed to construct a supervised fine-tuning dataset specific to the remote sensing domain. This step aims to create high-quality training data to guide the multimodal model in understanding remote sensing tasks and planning corresponding tool invocation steps. As shown in subsequent step S220, the supervised fine-tuning dataset described in this embodiment can be used to fine-tune existing, general-purpose multimodal models using instructions. Please refer to [reference needed]. Figure 2 The diagram shows in detail the internal process of step S100.

[0050] Specifically, step S100 includes step S110, which defines the metadata of the tools. In step S110, a structured knowledge base can be established for the specialized remote sensing analysis tools that the intelligent agent system can use. The metadata of each tool is typically defined in JSON or a similar format, including the tool's unique name (such as OpenVocabularyDetector), a natural language description of its function (such as a target detector that can detect objects of any specified category), definitions of input parameters (such as categories, a list of strings representing the target categories to be detected), and a description of the output format (such as returning a JSON file path containing the bounding box coordinates of all detected targets). The metadata of the tools provides the basis for the model to plan tasks, enabling the model to understand the function and invocation method of each tool.

[0051] After defining the tool knowledge base, step S120, which generates multiple types of training samples, can be executed. To enable the model to handle both complex tool calls and simple question-and-answer processes, step S120 generates various types of training samples. One important sample type is that which includes role-based dialogue sequence samples. Specifically, the generation process of role-based dialogue sequence samples is as follows: First, domain experts design a series of remote sensing task instructions that closely resemble real-world application scenarios and cover different levels of complexity, such as natural language task instructions to detect, count, and display the number of vehicles in this area. Then, using a generative model (such as a large language model), based on the tool metadata defined in step S110, the system automatically generates attribute descriptions of several tools that can be used in the training task, an execution plan for the training task, and several rounds of dialogue set according to the execution order of the training task, including tool call instructions corresponding to the model role and the execution results of the target tool corresponding to the tool role. For example, for the task described above, the generative model can plan a task execution plan that first calls an object detection tool to detect all 'vehicles', then calls an object counting tool to count the number, and finally calls a visualization tool to highlight the results. Following this execution order, the model generates several rounds of dialogue, including tool call instructions for the model role and the execution results of the target tools for the tool roles. The following is a JSON example of the training samples designed for the training task of detecting, counting, and displaying the number of vehicles in this area:

[0052] { image:<image file path>, task_type: detection_counting_visualization,

[0053] conversation:

[0054] [ { role: system,

[0055] content: You are a remote sensing multi-modal intelligent agent assistant, specializing in optical remote sensing image analysis tasks. Currently, you are equipped with the following three tools. Please select the most suitable tool for invocation according to the user's needs. List of available tools: 1. OpenVocabularyDetector (Open Vocabulary Detector) Function description: Detect objects of the user-specified category in an optical remote sensing image and generate bounding box localization data. This tool is based on open vocabulary detection technology and can identify multiple common object categories. Input parameters: file_image (string, required): The file path of the input image. categories (string, required): The category names to be detected, supporting Chinese or English, separated by commas for multiple categories. For example: car,truck. Output result: file_bboxes (string): The JSON file path containing the detected bounding box information, in the format { bboxes : [[x1,y1,x2,y2], ...], categories : [...]} . Usage restrictions: Only applicable to optical remote sensing images, not applicable to SAR images or other types of data. 2. ObjectCounter (Object Counter) Function description: Perform statistical analysis on the bounding box data in the detection results and accurately calculate the number of objects for each category. Input parameters: file_bboxes (string, required): The JSON file path containing the bounding box data, usually generated by the OpenVocabularyDetector tool. Output result: number (integer): The total number of detected objects. count_by_category (dictionary, optional): The quantity distribution counted by category. Usage restrictions: Requires a valid bounding box data file as input. 3. BboxVisualizer (Bounding Box Visualization Tool) Function description: Visualize and annotate the detected bounding boxes on the original image to generate an easy-to-understand visual result. Input parameters: file_image (string, required): The original image file path. file_bboxes (string, required): The JSON file path containing the bounding box data. output_path (string, optional): The save path for the visualization result. Output result: file_visualize (string): The file path of the annotated visualization image. annotated_image (image data, optional): The image data of the annotation result. Usage restrictions: Requires the original image and the corresponding bounding box data file. Invocation rules: 1. Carefully analyze the user's needs and select the most suitable tool sequence.2. Each tool call can use JSON format:<tool_call> { name : tool name, parameter name : parameter value, ...}<tool_call> 3. After the tool is invoked, you will receive the results returned by the tool, in the following format:<tool_return> { Output parameters: values, ...}<tool_return> 4. Depending on the complexity of the task, multiple tools can be combined to complete a comprehensive analysis. 5. Finally, a clear and complete analysis report is provided to the user.

[0056] { role: user, content: [ { type: text, text: Detects car targets in the image.},

[0057] { type: image, image:<image file path>},

[0058] { role: assistant,

[0059] Content: You want to identify cars in an image, so I'll begin the process. Based on currently available tools, I will use a three-step workflow: first, use an open-vocabulary detector to locate the cars; then, use a counter to accurately count them; and finally, generate an labeled image using a visualization tool. Step 1: Preliminary analysis using an open-vocabulary detector.<tool_call> { name : OpenVocabularyDetector , file_image : cache image.png}<tool_call>},

[0060] { role: tool, content: [ { type: text, text: The open vocabulary detector has completed the analysis and returned vehicle location data (file 'cache bboxes.json'), which can be used for subsequent processes.<tool_return> {file_bboxes : cache bboxes.json}<tool_return>} ]},

[0061] { role: assistant, content: The car target in the image has been successfully detected, and a bounding box data file has been generated. Now proceed to the second step: use a precise counting tool to verify the number of cars and ensure the reliability of the results.}<tool_call> { name : ObjectCounter , file_bboxes : cache bboxes.json}<tool_call>},

[0062] { role: tool, content: [{type: text, text: The counting tool has completed the statistics and returned the result: 7 car targets were detected, waiting for the visualization tool to process.<tool_return> { number : 7}<tool_return>} ]},

[0063] {role: assistant, content: The image has been confirmed to contain 7 car targets. Now proceeding to step three: the visualization tool will create an annotated image clearly showing the location of each target.}<tool_call> { name :BboxVisualizer , file_image : cache image.png , file_bboxes : cachebboxes.json}<tool_call>},

[0064] {role: tool, content: [{ type: text, text: The visualization tool has completed processing, all cars have been labeled, and the visualization results are stored in cache visualization.png.<tool_return> {file_visualize : cache visualization.png}<tool_return>},{ type: image,image:<path to the output image file>}]},

[0065] {role: assistant, content: The three-step analysis process has been completed: 1) Detect car targets using OpenVocabularyDetector; 2) Count the number of cars using ObjectCounter, confirming a total of 7 cars; 3) Generate visualization results using BboxVisualizer; Analysis summary: In the image you provided, a total of 7 car targets were detected. The bounding boxes of all targets have been labeled in the visualization image, and the result file is saved in 'cachevisualization.png'.} ]}.

[0066] In the training samples in the aforementioned JSON format, the attribute descriptions of several tools available for the training task are designed as system prompts (i.e., the content corresponding to the system role: system). This design facilitates the multimodal model in establishing the logical relationship between tool attributes and training task execution based on the content of the training samples. More importantly, to improve the generalization and adaptability of the trained multimodal model, a dynamic system prompt mechanism can be introduced when constructing training samples: for example, randomly sampling different tool subsets (including the number, type, and order of tools) for each training sample, and applying diverse description templates and synonym substitutions to generate diverse training task descriptions, tool attribute descriptions, etc. This gives each training sample a unique system prompt. Simultaneously, for the same task, different and reasonable task plans and corresponding tool call sequences (i.e., the dialogue sequence between role: assistant and role: tool) can be generated as positive samples. For example, for the training task of detecting, counting, and displaying the number of vehicles in this area, the following training sample content can also be designed:

[0067] {image:<image file path>,

[0068] task_type: vehicle_detection_counting_visualization_alt,

[0069] conversation: [

[0070] { role: system,

[0071] Content: You are a remote sensing multimodal intelligent agent assistant, specializing in vehicle analysis tasks in optical remote sensing images. You are currently equipped with the following two tools. Please select the most suitable tool based on your needs. Available Tools List: 1. AllInOneVehicleAnalyzer (All-in-One Vehicle Analyzer) Function Description: An integrated end-to-end tool that completes vehicle detection, accurate counting, and result visualization in one go. It integrates a detection model, counting module, and rendering engine internally, eliminating the need for intermediate file transfers. Input Parameters: image_file (string, required): The path to the optical remote sensing image file to be analyzed. vehicle_type (string, optional): Specifies the vehicle type, such as car, truck, or all. Defaults to car. output_dir (string, optional): The output directory, defaults to the current directory. Output Results: final_count (integer): The total number of vehicles of the specified type detected in the image. visualization_path (string): The path to the visualization image file with bounding boxes and count annotations. detailed_report (string, optional): The path to a detailed report file (JSON format) containing the coordinates of each vehicle's location. Usage limitations: Only applicable to optical images; performance may degrade in densely populated areas with more than 100 vehicles. 2. QualityEnhancer (Image Quality Enhancer) Function Description: Preprocesses the input image to enhance contrast and reduce blur, improving the accuracy of subsequent analysis. Input parameters: input_image (string, required): Path to the original image file. enhancement_type (string, optional): Enhancement type; options: contrast, sharpness, denoise, all. Default is all. Output results: enhanced_image (string): Path to the enhanced image file. quality_score (floating-point number): Image quality improvement score. Usage limitations: Primarily used for optical images with slight blur or insufficient contrast. Calling rules: 1. Carefully analyze user needs and select the most suitable tool. If available tools can complete the task in one step, prioritize integrated tools to improve efficiency. 2. Each tool call uses JSON format:<tool_call> { name : tool name, parameter name: parameter value, ...}<tool_call> 3. After the tool is invoked, you will receive the results returned by the tool, in the following format:<tool_return> { Output parameters: ...}<tool_return> 4. Finally, provide users with a clear and complete analysis report.

[0072] { role: user,

[0073] content: [{ type: text, text: Detects car targets in the image.},{ type: image,image:<image file path>} ]},

[0074] { role: assistant,

[0075] Content: You want to identify car targets in an image, which typically involves detection, counting, and visualization. Based on currently available tools, I have an efficient solution: using AllInOneVehicleAnalyzer. It can complete all tasks in one step, without multiple calls and intermediate file transfers. Now let's start calling the integrated tool for analysis:<tool_call> { name :AllInOneVehicleAnalyzer , image_file :cacheimage.png ,vehicle_type :car , output_dir : cache}<tool_call>},

[0076] { role: tool,

[0077] content: [ { type: text, text: The integrated vehicle analyzer has completed execution. Vehicle targets in the image were successfully detected, counted, and visualization results were generated.<tool_return> { final_count : 7,visualization_path : cache visualization.png , detailed_report : cachedetailed_report.json}<tool_return>},{ type: image, image: cachevisualization.png} ]},

[0078] { role: assistant,

[0079] Content: Analysis complete! By calling the integrated vehicle analyzer, I have completed the tasks you specified: Detection and statistical results: A total of 7 cars were detected in the image you provided. Output results: 1. Visualized and annotated image: Saved to cache visualization.png, all detected cars are marked with bounding boxes. 2. Detailed data report: Information containing the precise location of each car has been saved to cache detailed_report.json. This integrated tool completes the detection, counting, and visualization process in one go, with accurate and efficient results.

[0080] For the training task of counting and displaying the number of vehicles, the two training sample examples above provide different descriptions of tool attributes, selection of tool sets, task execution planning, and role-based dialogue history. This mechanism forces the multimodal model to learn to parse tool capabilities from dynamically changing descriptions and make correct task planning decisions, rather than memorizing fixed tool combinations, and further enables the model to gain zero-sample adaptation to new tools.

[0081] Understandably, the task planning embodied in this dialogue sequence simulates the thought process of an expert. Finally, to ensure the accuracy and optimality of the generated sequences, remote sensing experts validate and correct these automatically generated dialogue sequences, eliminating illogical or inefficient plans, thereby forming high-quality expert reasoning trajectories. By adopting this mode of combining large-scale model generation with expert validation, it is possible to efficiently construct large-scale, high-quality supervised fine-tuning datasets.

[0082] After the samples are generated, step S130 is executed to format the training samples, converting the generated samples into the standard format required for model training. (Refer to...) Figure 5This diagram illustrates a preferred training sample data structure. In this embodiment, each training sample is formatted as a JSON object, containing the identifier of the remote sensing image (e.g., file path), the task type, and a dialogue history named "conversation." This dialogue history is an array, where each element represents a round of dialogue and has two fields: "role" and "content." "Role" can be "system" (the intelligent agent system or toolkit), "user," "assistant" (the assistant, i.e., the model), or "tool." For example, system prompts are placed in the "content" of "role:system"; user instructions are placed in the "content" of "role:user"; the model's thinking, decision-making (e.g., task planning), and tool invocations are placed in the "content" of "role:assistant"; and the tool's execution results are placed in the "content" of "role:tool." Figure 5 As shown, the tool invocation instruction is encapsulated within a specific first structured tag, such as the tool invocation tag.<tool_call> Its content is a JSON string containing the tool name and parameters. When the tool executes, its returned result (such as a data file path or a piece of text information) is encapsulated in another specific second structured tag, such as the tool return tag.<tool_return> This is then placed within the `content` of `role:tool`. Through this structured dialogue format, the model can learn during training which tool call label should be generated in which context.<tool_call> And how to understand the tool's returned labels<tool_return> The information in the middle.

[0083] After data preparation is complete, the process proceeds to step S200, which involves training the multimodal model. This step aims to inject the domain knowledge built in step S100 into a general multimodal model base, making it an expert in the remote sensing field. Please refer to the relevant documentation. Figure 3 The diagram shows in detail the internal flow of step S200.

[0084] Step S200 first executes step S210, which constructs a domain-adapted image adapter as the model input interface. General-purpose multimodal models are typically designed to process common network images (such as three-channel RGB images in JPEG and PNG formats), while remote sensing images possess diverse modalities (e.g., synthetic aperture radar (SAR) images are single-channel, while hyperspectral images have hundreds of channels), large sizes, and include geographic coordinates, among other specialized characteristics. To address this domain difference, this embodiment adds a multimodal remote sensing image adapter to the front end of the pre-trained general-purpose multimodal model. (Refer to...) Figure 7The diagram illustrates the detailed structure of the image adapter. This adapter primarily performs two core functions, as described below. The first is modality unification and image normalization—receiving raw multimodal remote sensing images, performing modality recognition and adaptation conversion; for example, using principal component analysis to reduce the dimensionality of hyperspectral images to three channels; performing channel duplication on single-channel SAR images to form three-channel images; subsequently, scaling all images uniformly to the input size required by the model's visual encoder (e.g., 512x512 pixels), and finally outputting a normalized three-channel image for model use. The second is image metadata parsing and text description generation—the image adapter automatically parses the metadata of the raw images (such as sensor type, spatial resolution, geographic coordinates, etc.) in parallel and converts it into a structured natural language image description text. For example, generating: This is a C-band SAR image taken by the Sentinel-1 satellite in October 2023, with a spatial resolution of 10 meters. Through the above processing, the original remote sensing images are transformed into visual and text signals that can be understood by multimodal models, while retaining their key professional physical properties. The original data is kept for use by tools, laying the foundation for subsequent domain understanding.

[0085] Subsequently, supervised fine-tuning step S220 based on a dedicated supervised fine-tuning dataset is performed. In this step, the normalized image output from the image adapter in step S210 is input into the visual encoder of the multimodal model to generate visual features. Simultaneously, the text information containing image metadata output from the image adapter is concatenated with the user's natural language task instructions to form a complete text prompt, which is then input into the aforementioned multimodal model. Then, the entire multimodal model is supervised fine-tuned using the formatted supervised fine-tuning dataset constructed in step S100. As an optional implementation, to save computational resources and training time, parameter-efficient fine-tuning methods, such as low-rank adaptive techniques, can be used. Step S230 aims to achieve targeted injection of tool invocation capabilities; its training objective is to enable the model to learn and predict the next most appropriate lexical unit to generate given the current dialogue history (including images, instructions, and previous tool interactions). Since the training data contains a large number of tool invocation tags...<tool_call> and tools return tags<tool_return> By learning from these samples, the structure and model can internalize the ability to plan from user intent to specific tool invocation steps.

[0086] After the multimodal model training is completed, the next step, S300, is to deploy and run the intelligent agent system based on the multimodal model. Please refer to [reference needed]. Figure 4 and Figure 8 . Figure 8The architecture diagram of the intelligent agent system provided in this embodiment is shown. The system includes key components such as a multimodal remote sensing image adapter, a multimodal model as the intelligent hub, a tool knowledge base, and an image database.

[0087] The intelligent agent system begins its operation in step S310, receiving user input. The user uploads a remote sensing image and inputs a natural language command, such as detecting and counting cars in the image, through a front-end user interface. After receiving this input, the system first preprocesses the image using a multimodal remote sensing image adapter, generating a normalized image and image description text. This process is similar to the preprocessing process during the training phase. It is worth noting that the initial input from the user does not necessarily include a remote sensing image. The user can specify a particular remote sensing image object in the image database using natural language commands, or instruct the multimodal model to search the image database for a specific remote sensing image object.

[0088] Next, in model inference step S320, the system inputs the initial current context (including the normalized image, image description text, user task instructions, and attribute descriptions of several tools supporting task planning and tool invocation for the multimodal model) into the trained multimodal model for inference, to generate a task planning decision for the remote sensing task instructions, and incorporates this task plan into the current context of the multimodal model, for example, as short-term memory of the large model. In the above process, the attribute descriptions of the optional tools provided to the multimodal model can be provided to the multimodal model in the form of system prompts, similar to the attribute descriptions of various tools provided in the content corresponding to role:system in the aforementioned JSON format training samples. Those skilled in the art will anticipate that, in order to shorten the length of the current context for model processing, the attribute descriptions of the aforementioned tools can also be stored in a tool database connected to the multimodal model, and task planning and tool invocation decisions can be made based on retrieval-enhanced generation (RAG).

[0089] Next, in step S330, a loop for tool execution is performed based on task planning, and the call result is obtained in each iteration of tool invocation to drive task execution. Specifically, the above loop may include the following two steps: Step A, based on the current context of the multimodal model, the multimodal model generates the next operation instruction based on the tool call sequence task planning; Step B, the next operation instruction is parsed. If the next operation instruction is a call instruction for the target tool, the target tool is invoked, and the execution result of the target tool is updated to the current context, and then the process returns to step A; if the next operation instruction is a task completion instruction, the final execution result summarized and output by the multimodal model in the task completion instruction is obtained.

[0090] Specifically, during the execution of the above loop, if the next operation instruction is plain natural language text and the model determines that the task has been completed (e.g., a summary answer has been generated), then the instruction is considered a task completion instruction, and the loop terminates. If the instruction is a tool invocation instruction (i.e., it contains a tool invocation tag), then...<tool_call> If the command is executed, the system will perform a tool call parsing and execution step. The system will extract the tool name and parameters from the command and then call the corresponding tool from the tool library. Finally, in the context update step, the result returned by the tool after execution (e.g., a data file path, a value, or a status text) is encapsulated in the tool return label.<tool_return> The tool call instructions generated by the model in this round are added to the current context as input for the next iteration, thus returning to operation a) to continue execution.

[0091] The execution loop process of the above-mentioned tools based on the multimodal model in the intelligent agent system can be as follows: Figure 6 As shown below, taking the user-instructed detection and counting of cars in an image as an example, the specific process of task execution is similar to that illustrated by the two JSON training sample data above: After receiving contextual information including the image (and its text description), user instructions, and attribute descriptions of available tools, the multimodal model infers and first decides on the plan for executing the task. For example, based on the currently available tools, I will adopt a three-step process: first, use an open-vocabulary detector to locate the cars; then, use a counter to accurately count the number; finally, use a visualization tool to generate an labeled image, and then enter the work loop of tool invocation and response execution. The working process of this loop can be described as follows:

[0092] In the first iteration, the multimodal model, based on the task planning in the current context and the attribute description of the tool, generates the next operation instruction for detecting and counting cars in the image:<tool_call> { name :OpenVocabularyDetector , categories : automobiles}.

[0093] The system parses the instruction, calls a tool called OpenVocabularyDetector, and passes the car as a parameter.

[0094] After execution, the tool returns a JSON file path containing the bounding box coordinates of all detected cars, for example, results bboxes_123.json.

[0095] The system updates the current context with the tool's execution results. At this point, the context becomes a complete dialogue history that includes the user's initial command, the model's first invocation, and the tool's first return.

[0096] In the second iteration: the system inputs the updated context back into the above multimodal model.

[0097] The multimodal model infers and generates the next operation instruction based on the new information of the existing detection results in the context:<tool_call> { name : ObjectCounter , file_bboxes : results bboxes_123.json}.

[0098] The system parses and calls the ObjectCounter tool from the tool library. This tool reads the specified JSON file and calculates the number of bounding boxes, returning a result of 7. The system then updates the context with this result.

[0099] In the third iteration: the multimodal model receives context containing detection results and count values, determines that the core requirements of the task have been met, but decides to generate a visualization to provide a more user-friendly output. Therefore, it generates instructions for the next step:<tool_call> { name : BboxVisualizer , file_bboxes : resultsbboxes_123.json}.

[0100] The system invokes a visualization tool to generate a new image on which the detection boxes are drawn, and returns its path.

[0101] System update context.

[0102] In the fourth iteration: the multimodal model detects that the context already contains the detection results, count values, and visualization results, and determines that the entire task has been successfully completed. The multimodal model generates a task completion instruction, which is a summary natural language text: Detection and counting have been completed; there are a total of 7 cars in the image; visualization results have been generated.

[0103] The intelligent agent system has determined this to be the final answer, and thus terminates the iteration loop.

[0104] Finally, in step S340, the system integrates and outputs the results. The system integrates the final natural language summary, structured data (such as counting results), and visualization images according to a preset format. For example, the system edits and fills the corresponding sections of a preset H5 page template and outputs it to the user for H5 page display. As a preferred implementation, the entire intelligent agent system can preset multiple interaction nodes for the process of performing the above remote sensing tasks. When the execution process reaches a preset interaction node, the system can output the current execution status to the user, such as "The system is normalizing the remote sensing image... The model is thinking to generate task planning... Calling the target detection tool... Detection complete, 7 targets found...", which is pushed to the front-end user interface in real time in a streaming manner, thereby significantly improving the observability of the system and the user's interactive experience.

[0105] In summary, the method provided in this embodiment can automatically decompose, plan, and execute a complex remote sensing analysis task that requires multiple steps through a single natural language interaction with the user, and return a comprehensive result with rich graphics and text, significantly reducing the barrier to entry for using remote sensing technology.

[0106] In the above embodiments, when the multimodal model performs a task, the tool invocation and iteration context are managed by continuously appending the current context (or dialogue history), which is intuitive and easy to implement. However, when the task toolchain is long, the continuously appended dialogue history may lead to an overly verbose context, thus posing a challenge to the model's processing capabilities. This embodiment uses a structured world state object as the core of the iteration, making the task's context management more explicit and efficient.

[0107] This approach first defines a standardized world state data structure, typically in JSON format. This object serves as a centralized information hub, containing relevant information about the current task. As an example, a typical world state object might contain the following fields:

[0108] task_id: A unique identifier for the task.

[0109] user_instruction: The user's raw natural language instruction.

[0110] image_path: The path to the associated remote sensing image.

[0111] workspace: A directory path used to store intermediate files (such as detection results, temporary images).

[0112] detected_objects: A list that stores detailed information about detected objects (such as category, bounding box, and confidence level).

[0113] analysis_results: A dictionary used to store key results from the analysis (such as the number of targets, area statistics).

[0114] remaining_goals: A list representing the remaining sub-goals to be completed in order to finish the entire task.

[0115] history: A log that records all actions that have been performed.

[0116] Accordingly, the model's input and output need to be adapted. The model's input is no longer a lengthy dialogue history string, but a string representation of the current world state object. The model's output is also no longer contained within natural language.<tool_call> It's not a label, but a specific, structured action object. Action objects can have different types, for example:

[0117] {action_type: tool_call, tool_name: OpenVocabularyDetector, parameters: categories: automobile}: This indicates that a tool is invoked.

[0118] {action_type: update_state, field: analysis_results, key: car_count, value: 7}: This indicates that a specific field of the world state is directly modified.

[0119] {action_type: finish_task, summary: Task completed...}: This indicates that the task has ended and provides a summary.

[0120] Based on this, the workflow of the dynamic iterative loop execution engine is adjusted accordingly as follows: First, during the initialization phase, when a new task begins, the system initializes a world state object based on user input, specifically parsing the initial `remaining_goals` from the task instructions, such as `[detect_cars, count_cars]`. Then, at the start of iteration, the loop execution engine serializes the current world state object into a string and provides it as input to the multimodal large model. Subsequently, the model makes a decision, analyzing the current state, particularly `remaining_goals` and existing `analysis_results`, and then generates the most suitable action object. Afterward, the action execution engine parses this action object. If `action_type` is `tool_call`, the corresponding tool is invoked. After the tool completes its execution, its results (e.g., a list of detected objects) are not directly appended to the dialogue history. Instead, the execution engine uses them, according to preset logic, to update the corresponding fields in the world state object (e.g., populate the `detected_objects` list) and remove completed goals from `remaining_goals` (e.g., remove `detect_cars`). If `action_type` is `update_state`, then the world state object is modified directly. Finally, a state check is performed; the execution engine checks if the `remaining_goals` list is empty. If it is empty, or the model has generated a `finish_task` action, the loop terminates; otherwise, it returns to step 2 and begins a new iteration.

[0121] Taking the "detect and count" task as an example, the iterative process based on structured states can be described as follows:

[0122] The initial state is: {..., remaining_goals: [detect_cars, count_cars], detected_objects: [], analysis_results: {}}.

[0123] The model reads the state and finds that the primary target is detect_cars, generating the action: {action_type: tool_call, tool_name: OpenVocabularyDetector, ...}.

[0124] After the tool is executed, the engine populates the detected_objects field with the detection results and updates the status to: {..., remaining_goals: [count_cars], detected_objects: [...], ...}.

[0125] The model reads the new state, finds that the next target is count_cars, and generates an action: {action_type:tool_call, tool_name: ObjectCounter, ...}.

[0126] After the tool executes, it returns a count of 7. The engine updates this result to the analysis_results field and updates the status to: {..., remaining_goals: [], ..., analysis_results: {car_count: 7}}.

[0127] The model reads the new state, finds that `remaining_goals` is empty, and generates the action: `{action_type: finish_task, summary: There are 7 cars in the image.}`. The loop ends.

[0128] Understandably, this iterative approach based on structured states makes the task's contextual information clearer, more controllable, and easier for machines to process. This approach shifts the model's responsibility from continuing the dialogue to more precisely focusing on generating actions that change the current world state. This method is particularly effective for very long, multi-stage remote sensing analysis task chains that require rigorous state tracking and complex logical judgments.

[0129] In another embodiment shown, the dynamic task planning and tool execution loop described in the foregoing embodiments are further optimized by introducing support for parallel tool scheduling, which aims to significantly improve the execution efficiency of complex tasks.

[0130] Many complex remote sensing analysis tasks often involve multiple subtasks that can be executed independently. For example, a user might request: "Please count the number of buildings in the left half of the image and the number of vehicles in the right half, and perform cloud cover analysis on the entire image." In this request, the three subtasks of "counting buildings," "counting vehicles," and "analyzing cloud cover" are independent of each other and can theoretically be performed simultaneously.

[0131] To achieve parallel scheduling, this embodiment makes the following improvements to the system. First, the output format of the model is expanded. During the training phase, the dataset can contain some samples, and the output of the assistant role is no longer a single tool call label.<tool_call> Instead, it is a list or representation of a directed acyclic graph containing multiple tool call instructions. For example, the model can generate output in the following format: [<tool_call> { name : ObjectCounter , parameters : {region : left_half , category : building}}<tool_call> ,<tool_call> { name :ObjectCounter , parameters : { region : right_half , category : vehicle}}<tool_call> ,<tool_call> { name : CloudAnalysisTool , parameters : {}}<tool_call> The output format explicitly indicates the execution engine, and the three tool calls can be executed in parallel.

[0132] Secondly, the execution engine in step S330 is upgraded to a scheduler that supports parallelism. When the scheduler receives a list of multiple tool call instructions generated by the model, it performs the following operations: First, it performs dependency analysis. The scheduler checks the dependencies between these instructions; a simple dependency check method is to analyze their input and output parameters. If the input of instruction A requires the output of instruction B, then A depends on B. In this example, the inputs of the three tool calls depend only on the original image and are independent of each other. Second, it performs task distribution. For all instructions without prerequisite dependencies, the scheduler distributes them to different worker threads, processes, or computing nodes and starts their parallel execution. Third, it performs result synchronization and aggregation. The scheduler asynchronously waits for all parallel tools to complete execution, which is usually achieved through callback functions, events, or Future Promise mechanisms. When a tool completes, its result is temporarily stored; when all parallel tools have completed execution, the scheduler aggregates their respective results. Fourth, it performs context updates. For example, in the current context maintained as a world state, the scheduler updates all aggregated results to the dialogue context or the world state described in Example 3 at once before triggering the next round of inference for the model. In the current context maintained as a dialogue history, the scheduler organizes and updates the execution results of each parallel tool in the content field of the dialogue history.

[0133] By introducing a parallel scheduling mechanism, for complex remote sensing analysis requests that can be decomposed into multiple independent subtasks, the system can fully utilize multi-core processors or distributed computing resources, significantly reducing the total execution time that would otherwise require serial execution. For example, if each of the three subtasks mentioned above requires 10 seconds, serial execution would take a total of 30 seconds, while the ideal time for parallel execution is only 10 seconds (depending on the slowest task). This greatly improves the system's throughput and response speed, thereby enhancing the user experience, especially when handling large-scale, time-sensitive analysis tasks.

[0134] In summary, the core of the technical solution provided in this specification lies in constructing a three-tiered intelligent agent system:

[0135] The first layer is the tool execution layer. This layer encapsulates a series of professional remote sensing interpretation functions (such as open set target detection, panoramic segmentation, geographic information annotation, height map estimation, SAR image question answering, remote sensing physics simulation engine, DeepSeek, front-end scheduler, and other sensing and execution tools) into standardized, programmable microservices.

[0136] The second layer: Intelligent decision-making layer. This involves domain-adaptive modifications to the general multimodal large model. A visual encoder is added for multimodal image perception, utilizing visual information to assist the large model's decision-making. A dedicated training set containing pairs of (images, questions, and tool invocation instructions) is constructed to fine-tune the model. Essentially, this "instills" the model with a "domain instinct" of when and how to invoke tools, enabling it not only to understand remote sensing images and natural language but also to output structured task planning and tool invocation sequences.

[0137] The third level: System workflow. The entire system workflow is as follows: Receiving instructions: The system receives natural language instructions and remote sensing images input by the user; Understanding and planning: A finely tuned multimodal model acts as the "brain," deeply understanding the input and generating a structured decision containing task planning and tool invocation instructions; Scheduling and execution: The system's task execution engine parses this decision and sequentially, in parallel, or conditionally invokes the corresponding microservices in the tool execution layer; Integration and output: The system automatically integrates the intermediate outputs of various tools and returns the final results (such as visualized annotation maps, statistical reports, and analysis reports) to the user in an interactive front-end format.

[0138] Figure 9 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 9As shown, device 400 mainly consists of a communication interface 402, a user interface 404, a processor 406, and a data storage 408. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 410. The communication interface 402 enables device 400 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 402 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 402 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 402 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 402 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0139] User interface 404 includes receiving user input and providing output to the user. Therefore, user interface 404 may include input components such as a keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 404 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 404 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 400 may support remote access from other devices via communication interface 402 or another physical interface (not shown). User interface 404 may be configured to receive user input, the position and movement of which may be indicated by an indicator or cursor described herein. User interface 404 may also be configured as a display device for rendering or displaying text fragments.

[0140] Processor 406 may contain one or more general-purpose processors and / or special-purpose processors.

[0141] Data storage 408 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 406. Data storage 408 may include removable and non-removable components.

[0142] Processor 406 is capable of executing program instructions 418 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 408 to implement the various functions described herein. Data storage 408 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 400, enable device 400 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 418 by processor 406 may result in processor 406 using data 412. The processor 406 described above can execute program instructions to implement the remote sensing task processing method based on a multimodal model as described in one or more of the above embodiments.

[0143] In addition to the remote sensing task processing methods based on multimodal models described in the above-mentioned embodiments, this specification also provides a remote sensing task processing device based on a multimodal model. This device can be deployed independently within the aforementioned intelligent agent system, or it can be distributed among multiple functional modules included in the intelligent agent system according to its functions, to achieve specific interaction and control functions. The intelligent agent system includes a multimodal model and an image adapter; the intelligent agent system has preset attribute descriptions for one or more tools; the device includes:

[0144] An image adaptation unit is used to respond to a task command for an image input by a user terminal, to perform normalization processing on the image using the image adapter, and to extract its metadata information to form image description text.

[0145] The task execution unit is configured to, based on the normalized image, the image description text, the attribute descriptions of one or more tools, and the task instructions, invoke the multimodal model to perform inference, generate a task plan for the task instructions, and repeatedly execute the following steps until the task is completed or a preset loop limit is reached:

[0146] Step A: Based on the current context of the multimodal model, generate the next operation instruction using the multimodal model based on the task planning;

[0147] Step B: Parse the next operation instruction. If the next operation instruction is a call instruction for the target tool, then call the target tool and update the execution result of the target tool to the current context, and then return to step A; if the next operation instruction is a task completion instruction, then obtain the final execution result summarized and output by the multimodal model in the task completion instruction.

[0148] The output unit is used to integrate the final execution result and output it to the user terminal.

[0149] More preferably, the invocation parameters of the one or more tools are preset in the system prompts of the multimodal model; or, the invocation parameters of the one or more tools are stored in a tool knowledge base connected to the multimodal model.

[0150] More preferably, the output unit is further configured to output the current execution state of the intelligent agent system to the user terminal, so as to visualize the current execution state on the user terminal.

[0151] More preferably, the data structure of the current context maintains a role-based dialogue sequence when the task planning is performed;

[0152] The invocation instruction of the target tool is encapsulated in the dialogue sequence and in the content corresponding to the role of the multimodal model, and the execution result of the target tool is encapsulated in the dialogue sequence and in the content corresponding to the role of the tool library.

[0153] More preferably, the data structure of the current context maintains the world state when the multimodal model executes the task planning; the world state includes the identifier of the task instruction, the current execution result of the task planning, and the remaining execution target of the task planning.

[0154] More preferably, the multimodal model is obtained based on the following training method:

[0155] Construct a supervised fine-tuning dataset for the target domain; wherein, the training samples in the supervised fine-tuning dataset include the labels of training images, the description of training tasks, the attribute descriptions of several tools, and the role-based dialogue sequence; the dialogue sequence maintains the execution plan of the training task corresponding to the model role, and several rounds of dialogue sequence set according to the execution order of the training task, including the tool calling instructions corresponding to the model role and the execution results of the target tool corresponding to the tool role;

[0156] The training images are processed using the image adapter according to a preset standardization method.

[0157] Based on the training images processed according to the preset specifications and the supervised fine-tuning dataset, the pre-trained multimodal model is subjected to supervised fine-tuning to obtain the multimodal model.

[0158] More preferably, the construction of the supervised fine-tuning dataset in the remote sensing field includes:

[0159] Construct a tool knowledge base for the target domain, wherein the tool knowledge base contains meta-information of the tools;

[0160] Using a generative model, a description of the training task, attribute descriptions of the tools, and the role-based dialogue history are generated in the training samples based on the meta-information of the tools and the type of the training task.

[0161] The training samples are validated by experts to construct the supervised fine-tuning dataset.

[0162] More preferably, the step of using a generative model to generate a description of the training task, attribute descriptions of the tools, and the role-based dialogue history in the training samples based on the meta-information of the tools and the type of the training task includes:

[0163] Using the generative model, different training samples are generated for the same training task based on the meta-information of the tools. The different training samples differ in any one or more aspects of the description of the training task, the selection of the set of tools, their attribute descriptions, and the role-based dialogue history.

[0164] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0165] What those skilled in the art will understand is:

[0166] In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0167] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0168] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0169] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0170] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0171] This specification uses specific terms to describe embodiments thereof. For example, "an embodiment" and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "an embodiment" or "an alternative embodiment" mentioned twice or more in different places in this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples, without contradiction.

[0172] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A task processing method based on a multimodal model, characterized in that, The method is applied to an intelligent agent system, which includes a multimodal model and an image adapter; the intelligent agent system has pre-defined attribute descriptions for one or more tools; the method includes: In response to a task command for an image input from the user, the image adapter is used to normalize the image and extract its metadata information to form image description text. The normalization process includes: using PCA dimensionality reduction to reduce images with more than three channels to three channels, performing channel completion on images with fewer than three channels, and proportionally reducing images with a resolution exceeding a preset threshold. The original image before processing is retained for use by the tool. Based on the normalized image, the image description text, the attribute descriptions of one or more tools, and the task instructions, the multimodal model is invoked to perform inference, thereby generating a task plan for the task instructions and repeatedly executing the following steps until the task is completed or a preset loop limit is reached: Step A: Based on the current context of the multimodal model, generate the next operation instruction using the multimodal model based on the task planning; Step B: Parse the next operation instruction. If the next operation instruction is a call instruction for the target tool, then call the target tool and update the execution result of the target tool to the current context, and then return to step A; if the next operation instruction is a task completion instruction, then summarize the execution result of the target tool to output the final natural language execution result. The final execution results are integrated and output to the user terminal, wherein the final execution results include the final natural language execution results, as well as the structured data and visualization images generated by calling the target tool; The data structure of the current context maintains the world state when the multimodal model executes the task planning; the world state includes the identifier of the task instruction, the current execution result of the task planning, and the remaining execution target of the task planning.

2. The method according to claim 1, wherein the invocation parameters of the one or more tools are preset in the system prompts of the multimodal model; or, the invocation parameters of the one or more tools are stored in a tool knowledge base connected to the multimodal model.

3. The method according to claim 1, wherein during the execution of the method, the current execution state of the intelligent agent system is output to the user terminal so as to visually display the current execution state on the user terminal.

4. The method according to any one of claims 1 to 3, wherein the multimodal model is obtained through a supervised fine-tuning process as follows: Construct a supervised fine-tuning dataset for the target domain; among which, The training samples in the supervised fine-tuning dataset include the identifiers of training images, descriptions of training tasks, attribute descriptions of several tools, and role-based dialogue sequences. The dialogue sequences maintain the execution plans of the training tasks corresponding to the model roles, as well as several rounds of dialogue sequences set according to the execution order of the training tasks, including tool call instructions corresponding to the model roles and execution results of the target tools corresponding to the tool roles. The training images are subjected to a preset normalization process using the image adapter. Based on the pre-normalized training images and the supervised fine-tuning dataset, the pre-trained multimodal model is subjected to supervised fine-tuning to obtain the multimodal model.

5. The method according to claim 4, wherein constructing the supervised fine-tuning dataset for the target domain comprises: Construct a tool knowledge base for the target domain, wherein the tool knowledge base contains meta-information of the tools; Using a generative model, a description of the training task, attribute descriptions of the tools, and the role-based dialogue history are generated in the training samples based on the meta-information of the tools and the type of the training task. The training samples are validated by experts to construct the supervised fine-tuning dataset.

6. The method according to claim 5, wherein generating a description of the training task, attribute descriptions of the plurality of tools, and the role-based dialogue history in the training samples using a generative model based on the meta-information of the tools and the type of the training task comprises: Using the generative model, different training samples are generated for the same training task based on the meta-information of the tools. The different training samples differ in any one or more aspects of the description of the training task, the selection of the set of tools, their attribute descriptions, and the role-based dialogue history.

7. A task processing device based on a multimodal model, characterized in that, Applied to an intelligent agent system, the intelligent agent system includes a multimodal model and an image adapter; the intelligent agent system has pre-defined attribute descriptions for one or more tools; the device includes: An image adaptation unit is used to respond to task instructions for images input by the user terminal, perform normalization processing on the image using the image adapter, and extract its metadata information to form image description text; wherein, the normalization processing includes: processing images with more than three channels to three channels using PCA dimensionality reduction, performing channel completion on images with less than three channels, and proportionally reducing images with a resolution exceeding a preset threshold; and retaining the original image before processing for use by the tool; The task execution unit is configured to, based on the normalized image, the image description text, the attribute descriptions of one or more tools, and the task instructions, invoke the multimodal model to perform inference, generate a task plan for the task instructions, and repeatedly execute the following steps until the task is completed or a preset loop limit is reached: Step A: Based on the current context of the multimodal model, generate the next operation instruction using the multimodal model based on the task planning; Step B: Parse the next operation instruction. If the next operation instruction is a call instruction for the target tool, then call the target tool and update the execution result of the target tool to the current context, and then return to step A; if the next operation instruction is a task completion instruction, then summarize the execution result of the target tool to output the final natural language execution result. The output unit is used to integrate and output the final execution results to the user terminal, wherein the final execution results include the final natural language execution results, as well as the structured data and visualization images generated by calling the target tool for execution; The data structure of the current context maintains the world state when the multimodal model executes the task planning; the world state includes the identifier of the task instruction, the current execution result of the task planning, and the remaining execution target of the task planning.

8. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-6 by executing the executable instructions.

Images