Water conservancy field data processing method and equipment based on multi-modal intelligent agent and medium

Abstract

The invention discloses a water conservancy field data processing method and device based on a multi-modal intelligent agent and a medium, belongs to the technical field of big data processing, and aims to solve the technical problem of how to automatically and universally process water conservancy field data to adapt to new water conservancy analysis requirements. The technical scheme adopted by the invention is as follows: skill registration: allowing external registration of a new water conservancy data processing skill, receiving skill information in a JSON format through a RESTful API (Application Program Interface), and storing the skill information in a waterresource list in a relational database; skill classification; analyzing a skill result; decomposing an initial task; skill selection and parameter generation; skill calling and result analysis are carried out; analyzing data; performing task progress evaluation and iteration; and outputting a final result according to progress evaluation and iteration conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a method, device and medium for processing data in the water conservancy field based on a multimodal intelligent agent. Background Art

[0002] At present, data processing and analysis in the field of water conservancy projects mainly rely on professional software and manual operations, which can usually only process specific types of hydrological, hydraulic or water quality data and lack the ability to integrate multi-source data. For example, HEC-RAS software is mainly used for river hydraulic analysis, while the SWAT model focuses on basin hydrological process simulation. Although these professional tools perform well in their respective fields, it is difficult to achieve cross-domain data integration and comprehensive analysis.

[0003] In addition, existing artificial intelligence systems face many challenges in processing professional data in the field of water conservancy. These data usually include complex spatiotemporal information, nonlinear relationships, and multi-scale features, such as hydrological time series, water quality parameters, and geographic information system (GIS) data. Traditional machine learning methods have difficulty in fully capturing the complexity and interrelationships of these data.

[0004] The existing water conservancy system has the following defects:

[0005] ① The existing water conservancy project analysis system is difficult to flexibly process a variety of professional data formats, such as hydrological model output, remote sensing images, real-time monitoring data, etc.;

[0006] ② The analysis and understanding of water conservancy professional data often requires the participation of experts in the field, which makes it difficult to achieve automation and generalization;

[0007] ③ When processing water conservancy data, existing water conservancy systems often find it difficult to effectively combine data analysis results with the reasoning capabilities of large models, such as automatically generating water conservancy project planning recommendations or risk assessment reports;

[0008] ④ There is a lack of a common method to dynamically expand the system's data processing capabilities to adapt to new water conservancy modeling and analysis needs, such as newly developed water quality models or climate change impact assessment tools.

[0009] Therefore, how to realize automatic and universal processing of water conservancy data and adapt to new water conservancy analysis needs is a technical problem that needs to be solved urgently. Summary of the invention

[0010] The technical task of the present invention is to provide a water conservancy data processing method, equipment and medium based on a multimodal intelligent agent to solve the problem of how to automatically and universally process water conservancy data and adapt to new water conservancy analysis needs.

[0011] The technical task of the present invention is achieved in the following way: a water conservancy field data processing method based on a multimodal intelligent agent, the method is as follows:

[0012] Skill registration: Allow external registration of new water conservancy data processing skills, receive skill information in JSON format through the RESTful API interface, and store the skill information in the water_resource_skills table in a relational database (such as PostgreSQL) for subsequent call and management;

[0013] Skill classification: Extract the functional descriptions of all skills from the water_resource_skills table, use the natural language processing library NLTK to perform text preprocessing such as word segmentation, stop word removal, and lemma restoration, use the TF-IDF (term frequency-inverse document frequency) method to convert the preprocessed descriptions into numerical vectors, and generate category descriptions based on the clustering analysis results, and then generate a skill classification table; when the dimension of the numerical vector exceeds 1024, use the principal component analysis technology (PCA) to reduce the feature dimension;

[0014] Skill result parsing: Create a result parser class for each registered water conservancy data processing skill to parse text results, image results, and structured data;

[0015] Initial task decomposition: divide the user's water conservancy project-related problems into a series of sub-problems, and select the corresponding skill category for each sub-problem, and then obtain the sub-problem list and the corresponding skill category;

[0016] Skill selection and parameter generation: Build and fill prompt templates for each sub-question, and use the language model to process the prompt templates to obtain the selected skills and parameter values;

[0017] Skill calling and result parsing: Use the requests library to call the API of the selected skill, pass in the generated parameters, and parse the text, image, and structured data according to the output format of the skill to obtain the parsing results;

[0018] Data analysis: Use language models (such as GPT-4, Tongyi Qianwen, Wenxin Yiyan, etc.) to process prompts, generate Python code, use the exec() function to execute the code in a controlled environment, and capture the printout;

[0019] Task progress evaluation and iteration: Use the prompt template to evaluate whether the user's question has been fully answered; if not, propose the next sub-question to be solved and the required skill category, fill in information to the prompt template comprehensive function, and use the language model to process the prompt template to decide whether to continue iterating;

[0020] Based on progress evaluation and iteration, output the final result: After the task is completed, use the prompt template to integrate all information to generate a comprehensive and concise answer that directly answers the user's original question.

[0021] Preferably, the skill information in JSON format includes:

[0022] skill_name: skill name (string, such as "HEC-RAS River Channel Analysis", "SWAT Hydrological Simulation");

[0023] description: skill function description (string);

[0024] url: calling URL (string);

[0025] input_params: input parameter list; the input parameter list uses a JSON object array, each object includes the parameter name, type and description;

[0026] output_format: output format description; the input format description uses a JSON object, which includes data type and structure description.

[0027] Preferably, the clustering is as follows:

[0028] Use K-means algorithm for clustering, and set the maximum number of categories to 10;

[0029] The silhouette coefficient is calculated as follows:

[0030] For each sample point i, the details are as follows:

[0031] (1) Calculate a(i): the average distance from point i to other points in the same cluster;

[0032] (2) Calculate b(i): the average distance from point i to all points in the nearest other clusters;

[0033] (3) Calculate the silhouette coefficient s(i), specifically: calculate the average of the silhouette coefficients of all sample points to obtain the silhouette coefficient of the overall clustering; and evaluate the clustering effect of different K values ​​(2 to 10), and select the K value with the highest average silhouette coefficient as the optimal number of clusters.

[0034] Preferably, the generated category description is as follows:

[0035] Build the category description summary prompt template as follows:

[0036] Functional description of a set of related skills:

[0037] {Skill Description 1}

[0038] {Skill Description 2}

[0039] …

[0040] Based on the functional description, please generate a concise water conservancy engineering skill category name of no more than 10 words; and generate a detailed category description, including the main features and application scenarios of the corresponding category skills, of no more than 50 words;

[0041] For each cluster, fill in the corresponding category skill description into the prompt template in the format, and use a large language model (such as GPT-4) to generate the category name and detailed description for each category.

[0042] Preferably, the skill classification table is generated as follows:

[0043] Create the water_skill_categories table in the database with the following fields:

[0044] category_id: integer type, used as the primary key to uniquely identify each skill category;

[0045] category_name: string type, stores concise category names generated by clustering algorithms and large language models, such as "hydrological simulation", "water quality analysis", "flood control planning", etc.;

[0046] description: text type, storing detailed category descriptions generated by large language models, including the main features and application scenarios of the corresponding category skills, not exceeding 40 words;

[0047] Save the clustering results and generated category information to the water_skill_categories table as follows:

[0048] Assign a unique category_id to each cluster;

[0049] Use the generated concise category name as category_name;

[0050] Use the generated detailed description as description;

[0051] Add a category_id field (integer type) to the water_resource_skills table and update the category ID of each skill to correspond to the category_id in the water_skill_categories table.

[0052] More preferably, the parsed text result is specifically: when a text result is received, the corresponding result text is directly returned without any additional processing, that is, the text result is regarded as the final output and is used directly.

[0053] Preferably, the image analysis results are as follows:

[0054] After receiving the image result, use the image encoding library (such as base64) to convert the corresponding image into a transmittable format for subsequent processing;

[0055] Build a prompt template that contains the skill name, description, and calling parameters; the purpose of the prompt template is to provide the necessary context for the subsequent image analysis;

[0056] In the prompt template, clearly indicate the image that needs to be analyzed and ask for a detailed description, especially related to the skill function and calling parameters;

[0057] Pass the constructed hints and the transformed images together to a large multimodal model (e.g. GPT-4withvision) so that the model can understand the image and generate corresponding description text;

[0058] Returns the descriptive text generated by the model as a result of the analysis of the image.

[0059] Preferably, the structured data is parsed as follows:

[0060] When structured data is received, the entire data structure is traversed using a recursive method. During the traversal process, corresponding processing measures are taken for different data types, as follows:

[0061] (1) If a basic data type of string, number, or Boolean value is encountered, the string, number, or Boolean value is retained directly;

[0062] (2) If a list or dictionary is encountered, convert the list or dictionary to a pandas DataFrame to facilitate subsequent data processing and analysis;

[0063] After processing is complete, a modified JSON structure is generated, where any DataFrame part is replaced with a reference to the DataFrame in a format such as "DF_1" to facilitate subsequent references;

[0064] Use the JSON schema library to generate the schema of the corresponding data to describe the structure and type of the data;

[0065] Build a new prompt template. The prompt template content includes the skill name, description, calling parameters, and generated JSON schema. The prompt template is designed to guide the model to generate result descriptions.

[0066] Pass the prompts and structured data together to a large language model to generate a concise description of the results, highlighting the main features and possible insights of the data;

[0067] Returns a dictionary containing a description of the results and a DataFrame object (if any) for subsequent use and analysis.

[0068] An electronic device comprising: a memory and at least one processor;

[0069] Wherein, the memory stores a computer program;

[0070] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the water conservancy field data processing method based on multimodal intelligent agent as described above.

[0071] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the water conservancy field data processing method based on a multimodal intelligent agent as described above.

[0072] The water conservancy field data processing method, device and medium based on multimodal agent of the present invention have the following advantages:

[0073] (i) The present invention provides a RESTful API interface that allows external registration of new water conservancy data processing skills and can dynamically expand its capabilities to meet the evolving needs of water conservancy projects;

[0074] (ii) The present invention can flexibly process a variety of professional data formats and realize automated and universal analysis of water conservancy professional data;

[0075] (III) The present invention can effectively combine the results of water conservancy data analysis with large model reasoning capabilities, and dynamically expand data processing capabilities to adapt to new water conservancy analysis needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The present invention is further described below in conjunction with the accompanying drawings.

[0077] Attached Figure 1 This is a flowchart of the data processing method in the water conservancy field based on multimodal intelligent agents. DETAILED DESCRIPTION

[0078] The water conservancy field data processing method, device and medium based on multimodal intelligent agent of the present invention are described in detail below with reference to the drawings and specific embodiments of the specification.

[0079] Embodiment 1:

[0080] As attached Figure 1 As shown, this embodiment provides a water conservancy field data processing method based on a multimodal agent, and the method is specifically as follows:

[0081] S1. Skill registration: Allow external registration of new water conservancy data processing skills, receive skill information in JSON format through the RESTful API interface, and store the skill information in the water_resource_skills table in a relational database (such as PostgreSQL) for subsequent call and management;

[0082] S2. Skill classification: Extract the functional description of all skills from the water_resource_skills table, use the natural language processing library NLTK to perform text preprocessing such as word segmentation, stop word removal, and lemma restoration, use the TF-IDF (term frequency-inverse document frequency) method to convert the preprocessed description into a numerical vector, and generate a category description based on the clustering analysis results, and then generate a skill classification table; when the dimension of the numerical vector exceeds 1024, use the principal component analysis technology (PCA) to reduce the feature dimension;

[0083] S3, skill result parsing: Create a result parser class for each registered water conservancy data processing skill to parse text results, image results and structured data;

[0084] S4, initial task decomposition: divide the user's water conservancy project related problems into a series of sub-problems, and select the corresponding skill category for each sub-problem, and then obtain the sub-problem list and the corresponding skill category;

[0085] Prompt template:

[0086] User question: {user_question}

[0087] Available skill categories:

[0088] {skill_categories}.

[0089] Output format:

[0090] 1. Sub-problem: [Sub-problem description]

[0091] Skill Category: [selected skill category] 2....

[0093] Fill the user question into {user_question}, separate the category_name of all the data in the water_skill_categories table with commas, concatenate them into a string, and fill it into {skill_categories}. Use a large language model (such as GPT-4, Tongyi Qianwen, Wenxin Yiyan, etc.) to process the prompt and obtain a list of sub-questions and corresponding skill categories.

[0094] S5, skill selection and parameter generation: Build and fill prompt templates for each sub-question, and use the language model to process the prompt templates to obtain the selected skills and parameter values;

[0095] For each sub-problem decomposed, construct a prompt template:

[0096] User's original question: {original_question}

[0097] Current sub-question: {sub_question}

[0098] Available skills:

[0099] {skills_in_category}.

[0100] Please select the most appropriate hydraulic analysis skill to solve the current sub-problem and provide the specific parameter values ​​required to call the skill. Note that the unit and range of the parameter should be consistent with hydraulic engineering practice.

[0101] Output format:

[0102] Selected skill: [skill name]

[0103] parameter:

[0104] [Parameter 1 name]:[Parameter 1 value]

[0105] [Parameter 2 name]:[Parameter 2 value] ...

[0107] Fill in the prompt template:

[0108] Fill in the user's original question into {original_question};

[0109] Fill the generated sub-question into {sub_question};

[0110] According to the skill category corresponding to the sub-question, query all skills belonging to this category in the water_resource_skills table;

[0111] Format the retrieved skill information (including skill name and brief description) into a list and fill it into {skills_in_category}.

[0112] Use large language models (such as GPT-4, Tongyi Qianwen, Wenxin Yiyan, etc.) to process prompts and obtain selected skills and parameter values.

[0113] S6. Skill calling and result parsing: Use the requests library to call the API of the selected skill, pass in the generated parameters, and parse the text, image, and structured data according to the output format of the skill to obtain the parsing results;

[0114] S7. Data analysis: Use language models (such as GPT-4, Tongyi Qianwen, Wenxin Yiyan, etc.) to process prompts, generate Python code, use the exec() function to execute the code in a controlled environment, and capture the print output; the details are as follows:

[0115] If the result contains a DataFrame, construct the prompt template:

[0116] User question: {original_question}

[0117] Current sub-question: {sub_question}

[0118] Data description: {result_description}

[0119] Data columns: {dataframe_columns}

[0120] Please generate Python code to analyze this data to answer the current sub-question. The code should use the pandas library and print the results.

[0121] Fill the prompt template with information:

[0122] Fill in the user's original question into {original_question};

[0123] Fill the current sub-question into {sub_question};

[0124] Fill in the generated result description into {result_description};

[0125] Convert the DataFrame column name list into a string and fill it into {dataframe_columns}.

[0126] Use large language models (such as GPT-4, Tongyi Qianwen, Wenxin Yiyan, etc.) to process prompts and generate Python code.

[0127] Use the exec() function to execute code in a controlled environment and capture the printout.

[0128] S8, Task progress evaluation and iteration: Use the prompt template to evaluate whether the user's question has been fully answered; if not, propose the next sub-question to be solved and the required skill category, fill in the prompt template comprehensive function with information, and use the language model to process the prompt template to decide whether to continue iterating; the details are as follows:

[0129] Prompt template:

[0130] User's original question: {original_question}

[0131] Current sub-question: {sub_question}

[0132] Analysis result: {analysis_result}

[0133] Evaluate whether you have fully answered the user's question. If not, suggest the next sub-question to be solved and the skill category required.

[0134] Output format:

[0135] Task status: [Completed / Not Completed]

[0136] If not completed:

[0137] Next sub-question: [sub-question description]

[0138] Required skill category: [skill category]

[0139] Fill the prompt template with information:

[0140] Fill in the user's original question into {original_question};

[0141] Fill the current sub-question into {sub_question};

[0142] Fill the generated analysis results into {analysis_result}.

[0143] Use a large language model to process the prompt and decide whether to continue iterating.

[0144] S9. Output the final result based on the progress evaluation and iteration: After the task is completed, use the prompt template to integrate all the information and generate a comprehensive and concise answer that directly answers the user's original question; the details are as follows:

[0145] When the task is completed, use the prompt template to consolidate all the information:

[0146] User's original question: {original_question}

[0147] Solution process:

[0148] {sub_questions_and_results}

[0149] Based on the information above, please generate a comprehensive yet concise response that directly addresses the user's original question.

[0150] Fill the prompt template with information:

[0151] Fill in the user's original question in {original_question}.

[0152] Organize all sub-questions and their corresponding analysis results into structured text and fill in {sub_questions_and_results}.

[0153] Use a large language model to generate the final answer.

[0154] The skill information in JSON format in step S1 of this embodiment includes:

[0155] skill_name: skill name (string, such as "HEC-RAS River Channel Analysis", "SWAT Hydrological Simulation");

[0156] description: skill function description (string);

[0157] url: calling URL (string);

[0158] input_params: input parameter list; the input parameter list uses a JSON object array, each object includes the parameter name, type and description;

[0159] output_format: output format description; the input format description uses a JSON object, which includes data type and structure description.

[0160] The clustering in step S2 of this embodiment is specifically as follows:

[0161] S201-1, use K-means algorithm for clustering, and set the maximum number of categories to 10;

[0162] S201-2. Calculation of silhouette coefficient, as follows:

[0163] For each sample point i, the details are as follows:

[0164] (1) Calculate a(i): the average distance from point i to other points in the same cluster;

[0165] (2) Calculate b(i): the average distance from point i to all points in the nearest other clusters;

[0166] (3) Calculate the silhouette coefficient s(i), specifically: calculate the average of the silhouette coefficients of all sample points to obtain the silhouette coefficient of the overall clustering; and evaluate the clustering effect of different K values ​​(2 to 10), and select the K value with the highest average silhouette coefficient as the optimal number of clusters.

[0167] The description of the generated category in step S2 of this embodiment is as follows:

[0168] S202-1. Build a category description summary prompt template, as follows:

[0169] Functional description of a set of related skills:

[0170] {Skill Description 1}

[0171] {Skill Description 2}

[0172] …

[0173] Based on the functional description, please generate a concise water conservancy engineering skill category name of no more than 10 words; and generate a detailed category description, including the main features and application scenarios of the corresponding category skills, of no more than 50 words;

[0174] S202-2. For each cluster, fill in the corresponding category skill description into the prompt template according to the format, and use a large language model (such as GPT-4) to generate a category name and detailed description for each category.

[0175] The skill classification table generated in step S2 of this embodiment is specifically as follows:

[0176] S203-1. Create a water_skill_categories table in the database with the following fields:

[0177] category_id: integer type, used as the primary key to uniquely identify each skill category;

[0178] category_name: string type, stores concise category names generated by clustering algorithms and large language models, such as "hydrological simulation", "water quality analysis", "flood control planning", etc.;

[0179] description: text type, storing detailed category descriptions generated by large language models, including the main features and application scenarios of the corresponding category skills, not exceeding 40 words;

[0180] S203-2. Save the clustering results and generated category information into the water_skill_categories table, as follows:

[0181] ①Assign a unique category_id to each cluster;

[0182] ②Use the generated concise category name as category_name;

[0183] ③ Use the generated detailed description as description;

[0184] For example, a record in the water_skill_categories table might look like this:

[0185] category_id:1

[0186] category_name:"Hydrological simulation"

[0187] description:"This type of skills is mainly used to simulate and predict hydrological processes, including rainfall-runoff simulation, flood forecasting, groundwater flow analysis, etc.; they are usually based on hydrological principles and mathematical models, processing time series data and spatial distribution data, and providing important basis for water resources management and flood control decision-making;"

[0188] Add a category_id field (integer type) to the water_resource_skills table and update the category ID of each skill to correspond to the category_id in the water_skill_categories table.

[0189] The parsed text result in step S3 of this embodiment is specifically: when the text result is received, the corresponding result text is directly returned without any additional processing, that is, the text result is regarded as the final output and used directly.

[0190] The image analysis results in step S2 of this embodiment are as follows:

[0191] ① After receiving the image result, use the image encoding library (such as base64) to convert the corresponding image into a transmittable format for subsequent processing;

[0192] ②Build a prompt template, which contains the name, description and calling parameters of the skill; the purpose of the prompt template is to provide the necessary context for the subsequent image analysis;

[0193] ③ In the prompt template, clearly indicate the image that needs to be analyzed and ask for a detailed description, especially related to the skill function and calling parameters;

[0194] ④ Pass the constructed prompts and the converted images together to a large multimodal model (such as GPT-4withvision) so that the model can understand the image and generate corresponding description text;

[0195] ⑤Return the descriptive text generated by the model as an analysis of the image results.

[0196] The parsing of structured data in step S3 of this embodiment is specifically as follows:

[0197] (I) When receiving structured data, use a recursive method to traverse the entire data structure. During the traversal process, take corresponding processing measures for different data types, as follows:

[0198] (1) If a basic data type of string, number, or Boolean value is encountered, the string, number, or Boolean value is retained directly;

[0199] (2) If a list or dictionary is encountered, convert the list or dictionary to a pandas DataFrame to facilitate subsequent data processing and analysis;

[0200] (ii) After processing is complete, a modified JSON structure is generated, where any DataFrame part is replaced with a reference to the DataFrame in a format such as "DF_1" to facilitate subsequent references;

[0201] (3) Use the JSON schema library to generate the schema of the corresponding data to describe the structure and type of the data;

[0202] (iv) Construct a new prompt template, which includes the name, description, calling parameters, and generated JSON schema of the skill; the prompt template is intended to guide the model to generate result descriptions;

[0203] (v) Pass the prompts and structured data together to a large language model to generate a concise description of the results, focusing on the main features of the data and possible insights;

[0204] (vi) Returns a dictionary containing a description of the results and a DataFrame object (if any) for subsequent use and analysis.

[0205] Embodiment 2:

[0206] This embodiment also provides an electronic device, including: a memory and a processor;

[0207] Wherein, the memory stores computer-executable instructions;

[0208] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the water conservancy field data processing method based on multimodal intelligent agent in any embodiment of the present invention.

[0209] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0210] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0211] Embodiment 3:

[0212] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the water conservancy field data processing method based on a multimodal agent in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0213] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0214] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0215] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0216] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A water conservancy data processing method based on multimodal intelligent agent, characterized in that: The method is as follows: Skill registration: Allow external registration of new water conservancy data processing skills, receive skill information in JSON format through the RESTful API interface, and store the skill information in the water_resource_skills table in the relational database; Skill classification: Extract the functional descriptions of all skills from the water_resource_skills table, use the natural language processing library NLTK to perform text preprocessing such as word segmentation, stop word removal, and lemma restoration, use the TF-IDF method to convert the preprocessed descriptions into numerical vectors, and generate category descriptions based on the clustering analysis results, and then generate a skill classification table; when the dimension of the numerical vector exceeds 1024, use the principal component analysis technology to reduce the feature dimension; Skill result parsing: Create a result parser class for each registered water conservancy data processing skill to parse text results, image results, and structured data; Initial task decomposition: divide the user's water conservancy project-related problems into a series of sub-problems, and select the corresponding skill category for each sub-problem, and then obtain the sub-problem list and the corresponding skill category; Skill selection and parameter generation: Build and fill prompt templates for each sub-question, and use the language model to process the prompt templates to obtain the selected skills and parameter values; Skill calling and result parsing: Use the requests library to call the API of the selected skill, pass in the generated parameters, and parse the text, image, and structured data according to the output format of the skill to obtain the parsing results; Data analysis: Use language models to process prompts, generate Python code, use the exec() function to execute the code in a controlled environment, and capture printouts; Task progress evaluation and iteration: Use the prompt template to evaluate whether the user's question has been fully answered; if not, propose the next sub-question to be solved and the required skill category, fill in information to the prompt template comprehensive function, and use the language model to process the prompt template to decide whether to continue iterating; Based on progress evaluation and iteration, output the final result: After the task is completed, use the prompt template to integrate all information to generate a comprehensive and concise answer that directly answers the user's original question.

2. The water conservancy field data processing method based on multimodal agent according to claim 1 is characterized in that: Skill information in JSON format includes: skill_name: skill name; description: skill function description; url: calling URL; input_params: input parameter list; the input parameter list uses a JSON object array, each object includes the parameter name, type and description; output_format: output format description; the input format description uses a JSON object, which includes data type and structure description.

3. The water conservancy field data processing method based on multimodal agent according to claim 1 or 2 is characterized in that: The clustering is as follows: Use K-means algorithm for clustering, and set the maximum number of categories to 10; The silhouette coefficient is calculated as follows: For each sample point i, the details are as follows: (1) Calculate a(i): the average distance from point i to other points in the same cluster; (2) Calculate b(i): the average distance from point i to all points in the nearest other clusters; (3) Calculate the silhouette coefficient s(i), specifically: calculate the average of the silhouette coefficients of all sample points to obtain the silhouette coefficient of the overall clustering; and evaluate the clustering effect of different K values, and select the K value with the highest average silhouette coefficient as the optimal number of clusters.

4. The water conservancy field data processing method based on multimodal agent according to claim 3 is characterized in that: The generated categories are described as follows: Build the category description summary prompt template as follows: Functional description of a set of related skills: {Skill Description 1} {Skill Description 2} …… Please generate a concise water conservancy engineering skill category name based on the functional description, no more than 10 characters; Generate a detailed category description, including the main features and application scenarios of the corresponding category skills, no more than 50 words; For each cluster, the corresponding category skill description is filled into the prompt template in the format, and a large language model is used to generate the category name and detailed description for each category.

5. The water conservancy field data processing method based on multimodal agent according to claim 4 is characterized in that: Generate a skill classification table as follows: Create the water_skill_categories table in the database with the following fields: category_id: integer type, used as the primary key to uniquely identify each skill category; category_name: string type, stores the concise category name generated by the clustering algorithm and large language model; description: text type, storing detailed category descriptions generated by large language models, including the main features and application scenarios of the corresponding category skills, not exceeding 40 words; Save the clustering results and generated category information to the water_skill_categories table as follows: Assign a unique category_id to each cluster; Use the generated concise category name as category_name; Use the generated detailed description as description; Add a category_id field to the water_resource_skills table and update the category ID of each skill to correspond to the category_id in the water_skill_categories table.

6. The water conservancy field data processing method based on multimodal agent according to claim 5 is characterized in that: The specific results of parsing text are as follows: when a text result is received, the corresponding result text is directly returned without any additional processing, that is, the text result is regarded as the final output and is used directly.

7. The water conservancy field data processing method based on multimodal agent according to claim 6 is characterized in that: The results of image analysis are as follows: After receiving the image result, use the image encoding library to convert the corresponding image into a transmittable format; Build a prompt template that contains the skill name, description, and calling parameters; the purpose of the prompt template is to provide the necessary context for the subsequent image analysis; In the prompt template, clearly indicate the image that needs to be analyzed and ask for a detailed description, especially related to the skill function and calling parameters; Pass the constructed hints and the transformed images to a large multimodal model so that the model can understand the image and generate corresponding description text; Returns the descriptive text generated by the model as a result of the analysis of the image.

8. The water conservancy field data processing method based on multimodal agent according to claim 7 is characterized in that: Parsing structured data is as follows: When structured data is received, the entire data structure is traversed using a recursive method. During the traversal process, corresponding processing measures are taken for different data types, as follows: (1) If a basic data type of string, number, or Boolean value is encountered, the string, number, or Boolean value is retained directly; (2) If a list or dictionary is encountered, convert the list or dictionary to a pandas DataFrame; After processing is complete, a modified JSON structure is generated, where any DataFrame parts are replaced with references to that DataFrame, in the format of "DF_1"; Use the JSON schema library to generate the schema of the corresponding data to describe the structure and type of the data; Build a new prompt template, which includes the skill name, description, calling parameters, and generated JSON schema; The prompt template is designed to guide the model to generate result descriptions; Pass the prompts and structured data together to a large language model to generate a concise description of the results, highlighting the main features and possible insights of the data; Returns a dictionary containing a description of the results and a DataFrame object.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the water conservancy field data processing method based on multimodal intelligent agent as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the water conservancy field data processing method based on a multimodal intelligent agent as described in any one of claims 1 to 8.

Images