A method and device for generating an artificial intelligence resource portrait

Abstract

The application provides a method and device for generating an artificial intelligence resource portrait, the method comprising: obtaining an artificial intelligence resource to be optimized and an initial function description thereof, and constructing a test data set containing test questions and corresponding decision labels; performing multiple rounds of iterative optimization on the artificial intelligence resource according to the test data set, wherein each round of iteration comprises: processing the test questions by the artificial intelligence resource configured with the current function description through a master control model, generating a calling result, comparing the calling result with the decision labels to identify decision error samples, and generating a new function description based on the decision error samples for the next round of iteration; and when a preset stopping condition is met, generating a portrait of the artificial intelligence resource according to the function description corresponding to the optimal calling effect and the performance data in the test process, thereby achieving automatic perception and targeted optimization of the description of the resource capability boundary, and significantly improving the calling accuracy and stability of the model for various resources.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for generating artificial intelligence resource profiles. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have the ability to connect to external data and services. By calling various application programming interfaces or intelligent agent resources, the model can handle tasks such as real-time information query and complex business processing. In this process, the model mainly relies on the name and functional description text of the resource to understand its purpose and then determines whether to initiate a call operation. The quality of the resource description directly determines the accuracy of the model's intent recognition and the success rate of task execution.

[0003] However, in real-world applications, the descriptions of such resources are often written by developers based on their subjective understanding, resulting in semantic brevity and vague functional boundaries. They lack a systematic description of the resource's execution efficiency and capabilities. This inaccuracy makes it difficult for the model to accurately match the requirements when making decisions, easily leading to erroneous or missed calls, which wastes computing resources and reduces the overall service system's response efficiency.

[0004] Existing resource optimization methods mainly rely on manual post-analysis and adjustment, lacking a mechanism to simulate real-world usage scenarios for automated testing and iteration. This makes it difficult to quickly pinpoint the root cause of call failures and to quantify and evaluate the actual effects of description modifications. Therefore, there is an urgent need for a method that can automatically perceive resource capability boundaries and achieve self-optimization of descriptions based on test feedback, in order to improve the accuracy and stability of the model's calls to various resources. Summary of the Invention

[0005] This invention provides a method and apparatus for generating artificial intelligence resource profiles, which addresses the shortcomings of existing technologies such as the lack of automated testing and iteration mechanisms that simulate real-world scenarios, the inability to pinpoint the root causes of call failures, and the difficulty in quantifying and evaluating the effects of modifications.

[0006] This invention provides a method for generating artificial intelligence resource profiles, characterized by comprising: Obtain the AI ​​resources to be optimized and their initial functional descriptions, and construct a test dataset containing test questions and corresponding decision labels; The artificial intelligence resource is iteratively optimized in multiple rounds based on the test dataset. Each round of iteration includes: processing the test problem on the artificial intelligence resource configured with the current functional description through a preset master control model, generating a call result, comparing the call result with the decision label to identify decision error samples, and generating a new functional description based on the decision error samples for the next round of iteration. When the preset stopping conditions are met, a profile of the artificial intelligence resource is generated based on the functional description corresponding to the optimal call effect and the performance data during the test process.

[0007] According to the method for generating an artificial intelligence resource profile provided by the present invention, the step of constructing a test dataset containing test questions and corresponding decision labels includes: Configure a digital user group module, which contains several virtual users with different static profiles, dynamic recent information, and test domain characteristics; Test questions for the artificial intelligence resources are generated based on the virtual users in the digital crowd module; Send the test question to the AI ​​resource and receive the response from the AI ​​resource in response to the test question; A pre-trained response quality assessment model is used to perform a matching degree analysis on the test question and the response content. Based on the analysis results, it is determined whether the artificial intelligence resource has the ability to answer the test question. If it is determined that it has the ability, the decision label is marked as a positive example; if it is determined that it does not have the ability, the decision label is marked as a negative example.

[0008] According to the method for generating an AI resource profile provided by the present invention, the step of processing the test problem on the AI ​​resource configured with the current functional description through a preset master control model, generating a call result, and comparing the call result with the decision label to identify decision error samples includes: Register the artificial intelligence resources as external tools to the tool list of the master control model, and configure the current function description as the description information of the external tools; The test question is input into the master control model, and the master control model is monitored to see if it generates function call instructions for the artificial intelligence resource. If the function call instruction is generated, the call result is determined to be a call behavior; if the function call instruction is not generated, the call result is determined to be a non-call behavior. When the call result is a call behavior and the decision label is a negative example, or when the call result is a non-call behavior and the decision label is a positive example, the corresponding test problem is identified as the decision error sample.

[0009] According to the method for generating an artificial intelligence resource profile provided by the present invention, the step of generating a new functional description based on the decision error sample includes: The decision error samples are analyzed to extract a set of error causes that led to the decision errors called by the master control model; The initial function description and the set of error causes are input into a preset description optimization model to generate multiple candidate function descriptions. The score of each candidate function description is calculated according to a preset scoring rule. Then, with maximizing the score as a constraint, a new function description for the next iteration is selected from the candidate function descriptions.

[0010] According to the method for generating an artificial intelligence resource profile provided by the present invention, the step of calculating the score of each candidate function description according to a preset scoring rule includes: Determine the basic description score of the candidate function description; Determine the score for additional scope information compared to the basic description, which is used to measure the positive descriptive value of the breadth of functionality; Determine the range limitation penalty score relative to the basic description to measure the negative impact of incomplete enumeration on the breadth of functionality; Determine the score for excluded information relative to the basic description, which is used to measure the value of certainty brought by the process of elimination; Based on their respective weight coefficients, the basic description score, the additional range information score, and the exclusion information score are weighted and summed, and the weighted range limitation penalty score is subtracted to obtain the final score of the candidate function description.

[0011] According to the method for generating an artificial intelligence resource profile provided by the present invention, each iteration step further includes: performing attribution analysis and deduplication on the identified decision error samples to obtain key error features; generating supplementary test questions and corresponding decision labels based on the key error features; and adding the supplementary test questions and corresponding decision labels to the test dataset for subsequent rounds of invocation testing and verification.

[0012] According to the method for generating an AI resource profile provided by the present invention, the step of generating the AI ​​resource profile based on the functional description corresponding to the optimal call effect and the performance data during the testing process includes: Compare the call accuracy rates obtained from each round of iterations, and select the function description corresponding to the highest call accuracy rate as the target description for optimization; Based on the interaction logs during the test, the average first-character response time, average overall response time, and task execution success rate of the artificial intelligence resources were statistically analyzed. The target description, the capability boundary description, and the statistically obtained average first-word response time, average overall response time, and task execution success rate are structurally integrated to generate a profile of the artificial intelligence resource.

[0013] The present invention also provides an apparatus for generating artificial intelligence resource profiles, comprising: The module is used to acquire the AI ​​resources to be optimized and their initial functional descriptions, and to build a test dataset containing test questions and corresponding decision labels. The iteration module is used to perform multiple rounds of iterative optimization on the artificial intelligence resource based on the test dataset. Each round of iteration includes: processing the test problem on the artificial intelligence resource configured with the current functional description through a preset master control model, generating a call result, comparing the call result with the decision label to identify decision error samples, and generating a new functional description based on the decision error samples for the next round of iteration. The generation module is used to generate a profile of the artificial intelligence resource based on the functional description corresponding to the optimal call effect and the performance data during the test process when the preset stopping conditions are met.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for generating an artificial intelligence resource profile as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating an artificial intelligence resource profile as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for generating an artificial intelligence resource profile as described above.

[0017] This invention provides a method and apparatus for generating AI resource profiles. It constructs a test dataset containing test questions and corresponding decision labels, utilizes a master model to perform multiple rounds of closed-loop iterations on resources configured with current functional descriptions, compares the actual call results of the master model with preset labels to accurately identify decision-making errors, and automatically reconstructs the functional descriptions based on these errors for the next round of verification. Finally, it generates resource profiles based on the optimal call performance and test process data. This method, through a fully automated closed-loop testing and feedback mechanism, achieves automatic perception and targeted optimization of resource capability boundaries, quickly corrects fundamental defects leading to miscalls or missed calls, and verifies the optimization results with quantitative performance data, thereby significantly improving the model's accuracy and stability in calling various resources. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the method for generating AI resource profiles provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the structure of the artificial intelligence resource portrait generation device provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] In the current application development ecosystem of Large Language Models (LLMs), Function Calls have become a core means of extending model capabilities. Through this mechanism, developers can encapsulate external API interfaces, database query services, or agents with specific logic into tools, registering their names, parameter structures, and functional descriptions with the large model. When a user initiates a dialogue, the large model analyzes the user's intent in real time and determines whether to invoke a tool to assist in generating an answer based on the registered information. For example, in a smart assistant integrating multiple services, the system might simultaneously support tools such as "flight search," "hotel booking," and "weather service." The model relies entirely on the textual descriptions of these tools to decide which one to activate when faced with a user command.

[0024] However, existing semantic matching-based calling mechanisms require extremely high accuracy in descriptions, while in actual development, there is often a mismatch between descriptions and actual capabilities. Take a weather query tool called "Weather_Search" as an example. Initially, developers might simply describe it as "used to query city weather information." While this description is easy to understand, it doesn't define the tool's time frame or data dimensions. If the tool's underlying API actually only supports querying data for the "next 7 days" and doesn't include "historical weather," when a user asks "What was the temperature in Beijing yesterday?", the large model, based on the broad description, might misjudge that the tool has the ability to query historical data, thus initiating a call destined to fail. This kind of "illusionary" call caused by vague descriptions not only wastes the system's inference computing power and network resources but also severely degrades the user experience due to returned error information.

[0025] Faced with such problems, the current conventional solution mainly relies on manual intervention. Developers or operations personnel typically need to manually retrieve failed calls or unsatisfactory user feedback from the backend logs periodically after the service goes live, analyzing the root cause of each error. Once it's confirmed that the tool description misled the model, they will try to manually modify the description based on their personal experience, for example, changing the description of the weather tool mentioned above to "Query the weather forecast for the next 7 days; historical weather is not supported." After modification, the system is re-deployed to observe the effect. This traditional approach is not only passive, only fixing problems when they occur, but also lacks systematic regression testing. Manually modified descriptions often have unintended consequences—for example, emphasizing time limits might unexpectedly cause the model to hesitate to call the tool when faced with vague descriptions like "next week's weather," making it impossible to quantitatively assess the positive and negative impact on overall call accuracy while modifying the description.

[0026] To address the aforementioned issues, this invention proposes an automated, closed-loop method for generating AI resource profiles. Its core lies in constructing a digital testing environment capable of simulating real user interaction. Through a closed-loop process of "test generation - evaluation - analysis - optimization - iterative verification," it automatically identifies defects in the functional descriptions of AI resources and continuously optimizes these descriptions based on a data-driven approach. Ultimately, it generates a dynamic resource profile integrating precise functional descriptions, clearly defined capability boundaries, and key performance indicators. This profile can be used not only by resource providers to improve the resources themselves but also by resource users (such as master control models) as a basis for accurately invoking the resources.

[0027] Before describing the technical solutions of the embodiments of the present invention, the terms and concepts involved in the embodiments of the present invention will be explained illustratively.

[0028] Artificial intelligence resources refer to external capability modules that can be scheduled and executed through large model function call mechanisms, such as agents, application programming interfaces (APIs), and tools. These resources typically have well-defined functional boundaries and are used to extend the capabilities of large models for specific tasks.

[0029] Initial Functionality Description: This refers to the textual description provided by the developers when an AI resource is created or deployed, explaining its functionality and purpose. This description is usually brief and is the primary basis for large models to understand and decide whether to utilize the resource.

[0030] Test dataset: refers to the collection used in this invention to evaluate and optimize artificial intelligence resources, wherein each data point includes: a test question (generated by a digital population) and a corresponding decision label (indicating whether the question should be answered by the resource).

[0031] Master control model: refers to a large language model with the ability to invoke tools. It is responsible for determining whether a certain resource needs to be invoked based on the user's input question and the functional description of the registered artificial intelligence resources, and then executing the invocation decision.

[0032] Decision-making error samples: Samples in which the actual call decision is inconsistent with the preset decision label in the test dataset during the process of the master control model calling the test dataset. These mainly include two categories: incorrect calls (calling when it shouldn't be called) and missed calls (not calling when it should be called).

[0033] Optimal Invocation Performance: The state corresponding to the highest invocation accuracy achieved by the master control model in a certain round during multiple rounds of iterative optimization. The functional description and corresponding performance data in this state will be used to generate the final AI resource profile.

[0034] Digital User Group Module: This refers to the system module in this invention used to simulate a real user group. This module generates diverse test questions that closely resemble real-world scenarios by constructing a multi-dimensional profile for each virtual user, including static attributes (such as age and interests), dynamic status (such as recent interests), and test domain tags.

[0035] Answer quality assessment model: A trained discriminative model used to perform matching degree analysis on "test question - answer given by artificial intelligence resource" to determine whether the resource has the ability to answer the question correctly, thereby automatically labeling the test data with decision labels.

[0036] Description Optimization Model: An AI-based text generation and evaluation model that receives the current functional description and the error reasons obtained from the analysis of decision error samples, and generates new and more accurate functional description candidates with the goal of optimizing the preset comprehensive scoring function.

[0037] Basic description score: In the scoring function describing the optimization model, it is a basic score used to evaluate the consistency of the candidate function description with the initial function description in core semantics.

[0038] Additional scope information score: A positive scoring item in the scoring function, used to measure the value of the candidate description compared to the basic description by adding explicit functional scope information (such as time scope, spatial scope, included elements, etc.).

[0039] Exclusion Information Score: A positive scoring item in the scoring function, used to measure the certainty improvement value brought about by explicitly excluding certain features or content in the candidate description, such as excluding the UV index.

[0040] Scope narrowing penalty score: A negative scoring item in the scoring function, used to penalize the problem of unclear functional boundaries that may be caused by the use of incomplete enumeration (such as "including A, B, C, etc.") in the candidate description. The higher the value, the greater the uncertainty of the description.

[0041] Supplementary test questions: During the iterative optimization process, new test questions are generated specifically to further verify particular capability boundaries or error scenarios based on the root cause analysis results of decision error samples. These questions will be added to the test dataset to make subsequent tests more comprehensive.

[0042] The execution subject of the method in this embodiment of the invention is a computer device, specifically, it can be a server (including an independent physical server, a cloud server or a server cluster) or an AI resource management platform that integrates this function (such as an intelligent agent development platform, a MaaS service platform, etc.).

[0043] As the core control unit of the system, this execution entity has data processing, model scheduling and storage capabilities. It is responsible for coordinating the entire process of digital population data generation, main control model call testing, and subsequent root cause analysis and description optimization. Finally, it stores the generated AI resource profile in the database for downstream business use.

[0044] Figure 1 This is one of the flowcharts illustrating the method for generating an artificial intelligence resource profile provided in this embodiment of the invention, such as... Figure 1 As shown, the method includes: Step 101: Obtain the AI ​​resources to be optimized and their initial functional descriptions, and construct a test dataset containing test questions and corresponding decision labels.

[0045] The system first acquires the AI ​​resource to be optimized, such as an API providing weather query services, and reads its initial functional description, which is typically a concise text written by the resource provider. Then, the system initiates the process of building the test dataset. This process begins with configuring the digital population module. The digital population module contains a large number of predefined or algorithmically generated virtual users, each with a series of attributes covering relatively static identity characteristics, dynamically changing recent status information, and their respective test domain classification. Based on the core functional theme of the AI ​​resource under test, the system selects or combines virtual users from the digital population, driving it to generate a series of test questions related to the resource's functionality. These questions aim to cover various potential scenarios where the resource is invoked correctly, erroneously, or should have been invoked but was not.

[0046] Next, the system sends the generated test questions to the AI ​​resource one by one and collects the returned responses. For each "test question-response" pair, the system calls a pre-trained response quality assessment model for analysis. This model determines whether the AI ​​resource has the ability to correctly answer the question based on the correlation and completeness between the question semantics and the response content. If the model determines that the resource can effectively answer the question, it labels the test question as a "positive example," indicating that the question should be handled by this resource; if it determines that the resource cannot effectively answer the question, it labels it as a "negative example," indicating that the question should not be handled by this resource. Finally, the system integrates all labeled test questions and their corresponding decision labels to form a structured test dataset for subsequent iterative optimization processes.

[0047] Step 102: Perform multiple rounds of iterative optimization on the artificial intelligence resource based on the test dataset. Each round of iteration includes: processing the test problem on the artificial intelligence resource configured with the current functional description through a preset master control model, generating a call result, comparing the call result with the decision label to identify decision error samples, and generating a new functional description based on the decision error samples for the next round of iteration.

[0048] After entering the iterative optimization phase, the system initiates a closed-loop testing and correction process. This process aims to continuously calibrate the resource description text through a continuous feedback mechanism. At the beginning of each iteration, the system first determines the current version of the functional description to be used for testing. If it is the first iteration, the system uses the initial functional description of the AI ​​resource; if it is a subsequent iteration, it uses the updated version generated in the previous optimization. The system registers the AI ​​resource as an external tool to the master control model with tool invocation capabilities and configures the currently determined functional description as the tool's description information, thereby completing the deployment of the test environment.

[0049] After environment deployment, the system inputs the questions from the test dataset into the master control model one by one and monitors the model's inference output in real time. The master control model, based on the received question content and the configured function description, determines whether to invoke the corresponding AI resource to assist in generating the answer. The system records the master control model's specific behavior for each question as the invocation result, which is divided into invocation behavior (initiating a function call instruction) and non-invocation behavior (directly outputting a text reply). Subsequently, the system performs a comparison operation, matching the actual invocation result of each question with the preset decision labels in the test dataset. When the label indicates that an invocation should be made but the master control model does not, or when the label indicates that an invocation should not be made but the master control model initiates an invocation, the question is judged as not meeting expectations, and the system identifies and marks such questions as decision error samples.

[0050] After identifying erroneous decision-making samples, the system aggregates and analyzes these samples to extract potential causes of errors in the master control model's judgments. These causes might include overly broad descriptions leading to false triggers or missing key constraints resulting in missed triggers. Based on these error characteristics, the system automatically generates a new functional description containing corrective semantics. This new description aims to address decision-making biases exposed in the current round, such as adding explicit exclusion clauses or supplementing specific parameter range descriptions. The generated new description will be directly used to update the configuration information of AI resources, thereby initiating the next round of testing and verification until the system's performance reaches the predetermined standards.

[0051] Step 103: When the preset stopping conditions are met, generate a profile of the artificial intelligence resource based on the functional description corresponding to the optimal calling effect and the performance data during the testing process.

[0052] In step 103, the system continuously monitors the progress of the iteration process and terminates the optimization process when a preset stopping condition is met. The preset stopping condition is usually set based on the number of iteration rounds or the stability of performance indicators, such as reaching the maximum number of iteration rounds, or the improvement in the calling accuracy of the main control model falling below a certain threshold in multiple consecutive iterations.

[0053] Subsequently, the system retrospectively analyzed the call accuracy data recorded during each iteration. The system identified and selected the round with the highest call accuracy from all rounds, determining the function description used in that round as the optimal function description. Simultaneously, the system summarized and organized the performance data collected throughout the multi-round testing process. This data includes, but is not limited to, the average response time for AI resource processing requests, the success rate of task execution, and resource stability indicators under different loads.

[0054] Finally, the system structurally integrates the optimal functional description with the organized performance data to generate a complete profile of the AI ​​resource. This profile, as the final output of the optimization, comprehensively presents the resource's accurate capability boundaries and quantified performance in a machine-readable and human-understandable format after optimization and verification.

[0055] The AI ​​resource profiling generation method provided in this invention constructs a test dataset containing test questions and corresponding decision labels. It then uses a master control model to perform multiple closed-loop iterations on resources configured with current functional descriptions. The actual call results of the master control model are compared with preset labels to accurately identify decision-making errors. Based on these errors, the functional descriptions are automatically reconstructed for the next round of verification. Finally, a resource profiling is generated based on the optimal call performance and test process data. This method, through a fully automated closed-loop testing and feedback mechanism, achieves automatic perception and targeted optimization of resource capability boundaries. It can quickly correct fundamental defects leading to miscalls or missed calls and verifies the optimization results with quantitative performance data, thereby significantly improving the model's accuracy and stability in calling various resources.

[0056] Furthermore, in the specific implementation of constructing the test dataset, the system first initiates the configuration program for the digital user module. This module pre-configures multiple virtual user models with detailed personas, each persona consisting of three dimensions: static profile, dynamic situation, and testing domain. The static profile covers relatively fixed attributes such as age, gender, professional background, interests, and personality traits; the dynamic situation simulates the user's real-time state, such as recent hot topics, current emotional fluctuations, and changes in living environment; the testing domain is further subdivided into different scenarios such as normal functional consultation, ambiguous intent probing, abnormal boundary testing, and security compliance detection. Based on these rich combinations of dimensions, the system drives virtual users to generate a massive number of diverse test questions for the AI ​​resources under test, according to their respective persona backgrounds, thereby simulating complex and ever-changing interaction requests in the real world.

[0057] After generating the test questions, the system sends these test questions one by one to the AI ​​resources to be optimized, and simultaneously records the resource's response to each question. To establish a baseline for evaluation, the system calls a pre-trained response quality assessment model. This model performs semantic-level matching analysis on the input test questions and the responses returned by the resources, deeply determining whether the resource's response substantially solves the user's problem or whether its provided functionality truly covers the problem's requirements. Based on the analysis results, if the assessment model determines that the resource's response to a question is accurate and effective, indicating that the resource has the ability to handle such questions, the system marks the corresponding decision label as a positive example, meaning that the resource should be invoked in subsequent calls to the master model; conversely, if the response is determined to be invalid, off-topic, or explicitly indicates that it cannot be handled, the system marks the decision label as a negative example, indicating that the invocation should not be triggered. Through this process, the system automatically builds a high-quality test dataset containing questions, responses, and accurate decision labels, laying a solid data foundation for subsequent optimization iterations.

[0058] Further, after constructing the test dataset, the system enters the core call testing and error identification phase. First, the system formally registers the AI ​​resource to be optimized as a callable external tool in the master model's tool list, configuring the current version's function description text as the tool's official documentation, enabling the master model to perceive and understand the tool's purpose. Subsequently, the system inputs the test questions from the test dataset one by one into the master model and monitors the model's output stream in real time. During this process, the system does not focus on the natural language responses generated by the master model, but rather on whether it generates a specific function call instruction (FunctionCall) for the AI ​​resource. If the master model outputs a function call instruction conforming to the protocol specification, the system determines that the interaction has resulted in a call behavior; conversely, if the master model only generates a plain text response without triggering any tool instruction, it is determined to be a non-call behavior.

[0059] Next, the system compares the actual monitored call results with the preset decision labels in the test dataset one by one. When the system finds that the master control model has generated a call behavior, but the corresponding decision label is negative (i.e., it should not be called), it is judged as a wrong call error; when the master control model has not generated a call behavior, but the decision label is positive (i.e., it should be called), it is judged as a missed call error. The system uniformly identifies the test issues corresponding to these two types of situations as decision error samples and collects them as the input source for subsequent analysis.

[0060] Let's take an AI resource that has the function of "querying the weather for the next seven days" but does not support historical data queries as an example. Suppose there are two questions in the test dataset: Question A is "querying the weather in Beijing tomorrow", and its corresponding decision label is positive (should be called); Question B is "querying the weather in Beijing last month", and its label is negative (should not be called).

[0061] During testing, if the master control model correctly generates a function call instruction for problem A based on the current functional description, the actual result matches the label, and the sample is considered a correct decision. However, if, for problem B, the master control model mistakenly believes that the resource can retrieve historical data and initiates a call instruction, the system detects a conflict between the "call behavior" and the label "negative example" and classifies problem B as a "mistaken call" type of decision error sample. Alternatively, if, for problem A, the master control model fails to identify the tool's purpose due to an overly obscure description and does not initiate a call, the "non-call behavior" conflicts with the label "positive example," and the system classifies problem A as a "missed call" type of decision error sample. The system collects these identified specific error samples (such as problem B and the uncalled problem A), allowing the subsequent analysis module to infer that the description may lack a clear definition of the "time range" or have unclear semantic expressions.

[0062] Based on the identified decision-making error samples, the system initiates a description and optimization process.

[0063] First, conduct in-depth attribution analysis on these samples to identify common features that cause the master control model to make misjudgments, such as the lack of restrictions on certain specific parameters in the description or the ambiguity of keywords, thereby extracting a specific set of error causes.

[0064] Subsequently, the system inputs the initial functional description of the AI ​​resource, along with the extracted set of error causes, into a pre-defined description optimization model. This model, based on the generation capabilities of a large language model, specifically attempts to correct defects in the original description, generating multiple rewritten candidate functional descriptions at once. To select the optimal solution from these candidates, the system quantitatively scores each candidate description according to a pre-defined scoring rule. This scoring rule comprehensively considers dimensions such as the accuracy, completeness, and conciseness of the description.

[0065] Ultimately, the system selects the highest-scoring candidate from numerous candidates, using the maximum score as a constraint, and establishes it as the new functional description for the next round of iteration testing, thereby achieving the directed evolution and self-improvement of resource descriptions.

[0066] In the scoring phase of candidate function descriptions, the system employs a multi-dimensional quantitative evaluation mechanism to select high-quality descriptions that accurately cover the functional scope while effectively avoiding ambiguity. Specifically, the system first calculates a basic description score for each candidate description. This score primarily reflects the basic quality of the description text in terms of grammatical fluency and core keyword coverage, serving as a benchmark for scoring.

[0067] Building upon this, the system further introduces three correction items to refine the scoring. The first is the additional scope information score, which rewards descriptions that explicitly expand the breadth of functionality. For example, if a description explicitly states "supports major cities nationwide and overseas," compared to simply stating "query cities," this clear definition of spatial scope increases the discoverability of the functionality and thus receives a positive bonus.

[0068] Secondly, there is the scope limitation penalty score, which is used to penalize the risk of misjudgment of capabilities due to incomplete listing. For example, if the description uses an incomplete enumeration such as "including cities such as Beijing and Shanghai", the master control model may mistakenly believe that the tool only supports these two cities and ignore other supported cities. Such a statement will negatively limit the actual coverage of the function, so the system will calculate the corresponding penalty score and deduct it.

[0069] Finally, there's the exclusion information score, which rewards descriptions that clearly define capability boundaries through the process of elimination. For example, if a description specifically states "does not contain historical weather data" or "does not support feel temperature queries," this clear negative definition greatly eliminates the model's guesswork space and reduces the probability of incorrect calls, thus earning a higher positive weighting score.

[0070] Finally, the system calculates a weighted average of the scores based on pre-set weighting coefficients. The specific calculation logic is as follows: the basic description score, the weighted additional range information score, and the weighted exclusion information score are summed, and then the weighted range narrowing penalty score is subtracted. This comprehensive calculation yields a final score that accurately quantifies the overall performance of each candidate description in terms of accuracy, breadth, and boundary clarity, providing a scientific basis for subsequently selecting the optimal description.

[0071] The score can be maximized using the following formula: Score = base_score + L_range × W_range - P_limitation × W_limitation + L_exclusion × W_exclusion in: Score: Final score; base_score: Basic descriptive score; L_range: Additional range information score compared to the basic description (measures a positive description of the breadth of functionality); P_limitation: Compared to the basic description, the range limitation penalty (measures the negative impact of incomplete enumeration on the breadth of functionality); I_exclusion: Compared to the basic description, the score for excluded information (measures the value of certainty brought by the process of elimination). W_range: The weight of the positive descriptive terms affecting the breadth of functionality; W_limitation: Incomplete enumeration affects the weight; W_exclusion: Exclude information score weights.

[0072] For example, rate and rank the following descriptions: Basic description: Query weather-related information.

[0073] Description 1: Check the weather for the next seven days.

[0074] Description 2: Check the weather in major cities at home and abroad, including temperature, humidity, wind speed, etc.

[0075] Description 3: Query weather-related information for major cities at home and abroad over the past seven days, excluding wind chill, UV intensity, etc.

[0076] base_score=2, W_range=0.5, W_limitation=0.2, W_exclusion=0.5.

[0077] The scoring table is shown in Table 1 below.

[0078] Table 1 describe content L_range P_limitation I_exclusion Score Calculation Final score Ranking Basic Description Check weather-related information 0 0 0 2 + 0×0.5 - 0×0.2 + 0×0.5 2 4 Description 1 Check the weather for the next seven days 1 0 0 2 + 1×0.5 - 0×0.2 + 0×0.5 2.5 2 Description 2 Check the weather in major cities both domestically and internationally, including temperature, humidity, and wind speed. 1 1 0 2 + 1×0.5 - 1×0.2 + 0×0.5 2.3 3 Description 3 This search provides weather information for major cities both domestically and internationally over the past seven days, excluding factors such as perceived temperature and UV intensity. 2 0 1 2 + 2×0.5 - 0×0.2 + 1×0.5 3.5 1 Among them, L_range (range information): specifies the added time or space range (such as "the last seven days", "major cities at home and abroad").

[0079] P_limitation (limitation penalty): The negative impact on the breadth of functionality caused by incomplete enumeration (such as "including A, B, C, etc.").

[0080] I_exclusion (exclusion information): Clearly defines what is not included by using the exclusion method (e.g., "excluding X and Y"), increasing certainty.

[0081] Furthermore, each iteration includes a closed-loop process of dynamically expanding the test dataset. This process begins with a systematic attribution analysis of all decision-making error samples identified in the current round. The system invokes a pre-defined analysis model to diagnose each decision-making error sample, pinpointing the root cause that led the master control model to make an erroneous decision. The system then merges and deduplicates these root causes, extracting several representative key error features.

[0082] After identifying key error characteristics, the system generates targeted supplementary test questions. The generation process aims to construct test scenarios that directly reflect or trigger these specific error characteristics. For example, if the key error characteristic is "the resource description does not clearly define the time range," the system might generate questions like "query last year's historical weather data," which touches upon this boundary. For each newly generated supplementary test question, the system automatically or through an auxiliary model determines its corresponding decision label based on its relevance to the core functions of the AI ​​resource, clearly indicating whether the question should be answered by the target resource.

[0083] Finally, the system adds these newly generated supplementary test questions and their corresponding decision labels to the existing test dataset. The expanded dataset will be used for call verification of the master control model in the next round or subsequent iterations of testing. This mechanism ensures that the test dataset continuously evolves as the optimization process deepens, more comprehensively covering the discovered capability boundaries and error scenarios, thereby driving subsequent functional description optimizations to be more targeted and effective.

[0084] Finally, the step of generating a profile of the artificial intelligence resource based on the functional description corresponding to the optimal call effect and the performance data during the testing process is specifically implemented through the following process: The system first performs performance backtesting and selects the optimal description. It accesses data storing all iteration records, extracts and compares the call accuracy values ​​calculated in each iteration. Through comparison, the system identifies the iteration that achieved the highest call accuracy. Subsequently, the system determines the functional description used in that iteration as the final target description for this round of optimization. This target description represents the text version that, after multiple rounds of validation, can guide the master control model to make the most accurate call decisions.

[0085] Next, the system extracts and statistically analyzes performance data. The system retrieves detailed interaction logs recorded throughout the multi-round testing process. Based on these logs, the system performs the following calculations: First, it calculates the average time interval between receiving a request and the return of the first valid character from the AI ​​resource, obtaining the average first-character response time. Second, it calculates the average time interval between receiving a request and returning the complete final result, obtaining the average overall response time. Finally, it calculates the percentage of successful task executions (i.e., the resource returned a valid and correct response) out of the total number of tests, determining the task execution success rate.

[0086] Finally, the system performs structured profiling generation. The system integrates the target description obtained in the preceding steps, the capability boundary description verified and summarized during testing, and the calculated average first-word response time, average overall response time, and task execution success rate, according to a predefined data structure. This data structure typically covers multiple dimensions such as functional definitions, capability boundaries, and performance metrics. After integration, the system outputs this structured data, which is the final profiling of the AI ​​resource. This profiling, in a standardized form, comprehensively and quantitatively depicts the resource's optimized and accurate functional and performance characteristics.

[0087] The apparatus for generating an artificial intelligence resource profile provided in the embodiments of the present invention will be described below. The apparatus for generating an artificial intelligence resource profile described below can be referred to in correspondence with the method for generating an artificial intelligence resource profile described above.

[0088] This invention provides an apparatus for generating artificial intelligence resource profiles, see [link to apparatus]. Figure 2 ,include: Module 210 is used to obtain the artificial intelligence resources to be optimized and their initial functional descriptions, and to build a test dataset containing test questions and corresponding decision labels. The iteration module 220 is used to perform multiple rounds of iterative optimization on the artificial intelligence resource based on the test dataset. Each round of iteration includes: processing the test problem on the artificial intelligence resource configured with the current functional description through a preset master control model, generating a call result, comparing the call result with the decision label to identify decision error samples, and generating a new functional description based on the decision error samples for the next round of iteration. The generation module 230 is used to generate a profile of the artificial intelligence resource based on the functional description corresponding to the optimal calling effect and the performance data during the test process when the preset stopping conditions are met.

[0089] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a method for generating an artificial intelligence resource profile. This method includes: acquiring the artificial intelligence resource to be optimized and its initial functional description, and constructing a test dataset containing test questions and corresponding decision labels; performing multiple rounds of iterative optimization on the artificial intelligence resource based on the test dataset, wherein each round of iteration includes: processing the test question on the artificial intelligence resource configured with the current functional description using a preset master control model, generating a call result, comparing the call result with the decision labels to identify decision error samples, and generating a new functional description based on the decision error samples for the next round of iteration; when a preset stopping condition is met, generating a profile of the artificial intelligence resource based on the functional description corresponding to the optimal call effect and the performance data during the test process.

[0090] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for generating an artificial intelligence resource profile provided by the above methods. The method includes: acquiring the artificial intelligence resource to be optimized and its initial functional description, and constructing a test dataset containing test questions and corresponding decision labels; performing multiple rounds of iterative optimization on the artificial intelligence resource based on the test dataset, wherein each round of iteration includes: processing the test question on the artificial intelligence resource configured with the current functional description through a preset master control model, generating a call result, comparing the call result with the decision label to identify decision error samples, and generating a new functional description based on the decision error samples for the next round of iteration; when a preset stopping condition is met, generating a profile of the artificial intelligence resource based on the functional description corresponding to the optimal call effect and the performance data during the test process.

[0092] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for generating an artificial intelligence resource profile provided by the methods described above. The method includes: acquiring an artificial intelligence resource to be optimized and its initial functional description, and constructing a test dataset containing test questions and corresponding decision labels; performing multiple rounds of iterative optimization on the artificial intelligence resource based on the test dataset, wherein each round of iteration includes: processing the test question on the artificial intelligence resource configured with the current functional description through a preset master control model, generating a call result, comparing the call result with the decision label to identify decision error samples, and generating a new functional description based on the decision error samples for the next round of iteration; when a preset stopping condition is met, generating a profile of the artificial intelligence resource based on the functional description corresponding to the optimal call effect and the performance data during the test process.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating AI resource profiles, characterized in that, include: Obtain the AI ​​resources to be optimized and their initial functional descriptions, and construct a test dataset containing test questions and corresponding decision labels; The artificial intelligence resource is iteratively optimized in multiple rounds based on the test dataset. Each round of iteration includes: processing the test problem on the artificial intelligence resource configured with the current functional description through a preset master control model, generating a call result, comparing the call result with the decision label to identify decision error samples, and generating a new functional description based on the decision error samples for the next round of iteration. When the preset stopping conditions are met, a profile of the artificial intelligence resource is generated based on the functional description corresponding to the optimal call effect and the performance data during the test process.

2. The method according to claim 1, characterized in that, The step of constructing a test dataset containing test questions and corresponding decision labels includes: Configure a digital user group module, which contains several virtual users with different static profiles, dynamic recent information, and test domain characteristics; Test questions for the artificial intelligence resources are generated based on the virtual users in the digital crowd module; Send the test question to the AI ​​resource and receive the response from the AI ​​resource in response to the test question; A pre-trained response quality assessment model is used to perform a matching degree analysis on the test question and the response content. Based on the analysis results, it is determined whether the artificial intelligence resource has the ability to answer the test question. If it is determined that it has the ability, the decision label is marked as a positive example; if it is determined that it does not have the ability, the decision label is marked as a negative example.

3. The method according to claim 1, characterized in that, The process involves using a preset master control model to process the test problem using the artificial intelligence resources configured with the current function description, generating a call result, and comparing the call result with the decision label to identify decision error samples, including: Register the artificial intelligence resources as external tools to the tool list of the master control model, and configure the current function description as the description information of the external tools; The test question is input into the master control model, and the master control model is monitored to see if it generates function call instructions for the artificial intelligence resource. If the function call instruction is generated, the call result is determined to be a call behavior; if the function call instruction is not generated, the call result is determined to be a non-call behavior. When the call result is a call behavior and the decision label is a negative example, or when the call result is a non-call behavior and the decision label is a positive example, the corresponding test problem is identified as the decision error sample.

4. The method according to claim 1 or 3, characterized in that, The step of generating a new functional description based on the decision error samples includes: The decision error samples are analyzed to extract a set of error causes that led to the decision errors called by the master control model; The initial function description and the set of error causes are input into a preset description optimization model to generate multiple candidate function descriptions. The score of each candidate function description is calculated according to a preset scoring rule. Then, with maximizing the score as a constraint, a new function description for the next iteration is selected from the candidate function descriptions.

5. The method according to claim 4, characterized in that, The step of calculating the score of each candidate function description according to the preset scoring rules includes: Determine the basic description score of the candidate function description; Determine the score for additional scope information compared to the basic description, which is used to measure the positive descriptive value of the breadth of functionality; Determine the range limitation penalty score relative to the basic description to measure the negative impact of incomplete enumeration on the breadth of functionality; Determine the score for excluded information relative to the basic description, which is used to measure the value of certainty brought by the process of elimination; Based on their respective weight coefficients, the basic description score, the additional range information score, and the exclusion information score are weighted and summed, and the weighted range limitation penalty score is subtracted to obtain the final score of the candidate function description.

6. The method according to claim 1, characterized in that, Each iteration also includes the following steps: Attribution analysis and deduplication are performed on the identified decision-making error samples to obtain key error features; Based on the key error characteristics, generate supplementary test questions and corresponding decision labels; The supplementary test questions and their corresponding decision labels are added to the test dataset for use in subsequent rounds of testing and verification.

7. The method according to claim 1, characterized in that, The step of generating a profile of the artificial intelligence resource based on the functional description corresponding to the optimal calling effect and the performance data during the testing process includes: Compare the call accuracy rates obtained from each round of iterations, and select the function description corresponding to the highest call accuracy rate as the target description for optimization; Based on the interaction logs during the test, the average first-character response time, average overall response time, and task execution success rate of the artificial intelligence resources were statistically analyzed. The target description, the capability boundary description, and the statistically obtained average first-word response time, average overall response time, and task execution success rate are structurally integrated to generate a profile of the artificial intelligence resource.

8. A device for generating artificial intelligence resource portraits, characterized in that, include: The module is used to acquire the AI ​​resources to be optimized and their initial functional descriptions, and to build a test dataset containing test questions and corresponding decision labels. The iteration module is used to perform multiple rounds of iterative optimization on the artificial intelligence resource based on the test dataset. Each round of iteration includes: processing the test problem on the artificial intelligence resource configured with the current functional description through a preset master control model, generating a call result, comparing the call result with the decision label to identify decision error samples, and generating a new functional description based on the decision error samples for the next round of iteration. The generation module is used to generate a profile of the artificial intelligence resource based on the functional description corresponding to the optimal call effect and the performance data during the test process when the preset stopping conditions are met.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating an artificial intelligence resource profile as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for generating an artificial intelligence resource profile as described in any one of claims 1 to 7.

Images