Multi-agent perception and cognition generation and API (Application Program Interface) calling multi-modal evaluation method

Abstract

The invention discloses a multi-agent perception and cognition generation and API (Application Program Interface) calling multi-modal evaluation method. The method comprises the following steps of: 1, downloading and deploying a standardized data set to the local; 2, evaluating the perception ability of the multi-agent, including perceiving ten sub-tasks, and calculating scores by using the single-graph single-question accuracy and the single-graph double-question full-match rate as perception indexes; 3, calculating scores by adopting the single-graph single-question accuracy rate and the single-graph double-question full-match rate as cognitive indexes, and outputting MME basic scores in combination with the perception scores; 4, generating and evaluating a multi-agent picture, extracting a target keyword, detecting a to-be-evaluated image to obtain an actual object category set, and calculating an image element matching score in combination with matching of the target keyword and an object category; 5, executing multi-agent API calling validity evaluation, and outputting an API calling matching degree score; and 6, multi-dimensional scores are fused, a comprehensive evaluation result is output, and evaluation of the multi-modal multi-agent large model is realized.

Description

Technical Field

[0001] This invention addresses the shortcomings of current methods in comprehensively reflecting the performance of MLLMs, the lack of a comprehensive evaluation, and the limitations of the MME standard. As a multimodal evaluation benchmark, MME only covers the overall performance ranking of the model and lacks correlation with actual generated results. Furthermore, current evaluation methods lack function call evaluation, fail to quantify the effectiveness of agent interface calls, and cannot reflect the execution accuracy of cross-modal tasks. Therefore, this invention proposes a precise, comprehensive, multimodal, and multidimensional multi-agent large model evaluation method. Background Technology

[0002] With the rapid development of multimodal and multi-agent architectures, the quality assessment of these architectures has become a core pain point restricting the practical application of the technology. However, existing assessment methods have many key shortcomings and still face numerous thorny issues. Modal assessment is fragmented, often judging the semantic rationality of text or the quality of images in isolation, failing to effectively integrate the assessment of multimodal collaborative capabilities. Core dimensions such as the consistency between text descriptions and image content, and the correlation between cognitive reasoning results and perceptual information are often overlooked. While Multimodal Evaluation Metrics (MME) can rank the overall performance of models, it lacks a practical implementation plan adapted to the actual generated results of agents and does not incorporate fine-grained image assessment methods such as object detection, making it difficult to accurately reflect the actual quality of generated content. Function call assessment is significantly lacking, failing to quantify the effectiveness of multi-agent interface calls, such as whether core modules like image generation and text translation are correctly called, and thus failing to objectively reflect the execution accuracy of cross-modal tasks. The degree of automation is low, relying heavily on manual scoring or single-dimensional indicators, lacking automated assessment schemes that can integrate multi-dimensional indicators, resulting in highly subjective and inefficient assessment results. Summary of the Invention

[0003] To address the problems of existing technologies, this invention provides a multi-agent perceptual and cognitive generation and API call multimodal evaluation method. This method combines multiple dimensions to construct a multimodal multi-agent large model evaluation index. Considering that using only the Multi-Agent Model (MME) as a multimodal evaluation benchmark lacks evaluation of the actual generated results and function calls of the agents, this invention utilizes object detection to evaluate the completeness of generated image elements. Furthermore, it uses the mapping relationship between preset function vectors and actual API call vectors of the agents to evaluate the collaborative effect between agents within the multi-agent system and the effectiveness of function calls, objectively reflecting the execution accuracy of cross-modal tasks.

[0004] This invention is achieved through the following method: a multi-agent perception and cognition generation and API call multimodal evaluation method, the steps of which are as follows: Step 1) Using a publicly available standardized dataset, download the MME dataset and deploy it locally. Input the image generation command, which involves multiple objects. 1.1) Download the open-source MME dataset. This dataset comprehensively evaluates the perceptual and cognitive abilities of MLLM, with the two abilities supporting each other. The dataset is diverse; some images are sampled from public datasets, but only the images themselves are used, without relying on the original annotations. It is supplemented with a large number of manually taken images, including handwritten arithmetic problems, photos of everyday scenes, and images generated by the diffusion model, further reducing the risk of leakage.

[0005] 1.2) Load the perceptual ability dataset. This dataset is mainly divided into three categories: coarse-grained recognition, fine-grained recognition, and OCR. Coarse-grained recognition is used to evaluate the recognition of basic attributes of common objects, including four sub-tasks; fine-grained recognition is used to evaluate the professional recognition of content in specific domains, including five sub-tasks; the OCR task is used to evaluate image-to-text recognition capabilities, selecting only simple text samples, containing 20 images and 40 sets of instruction-answer pairs.

[0006] 1.3) Load the cognitive ability dataset. This mainly includes four sub-tasks: commonsense reasoning, numerical computation, text translation, and code reasoning. All images for the cognitive tasks were taken manually, and the instruction-answer pairs were designed manually to ensure no data leakage.

[0007] 1.4) Obtain image generation instructions. Input the image generation instructions, which mainly include generating multiple elements within an image. Load the object detection model to support fast detection of object categories in the image.

[0008] Step 2) Perform a multi-dimensional capability assessment based on the MME standard to evaluate the perception capabilities of multiple agents, including ten perception sub-tasks, including object existence recognition, object counting, object position recognition, color recognition, movie poster recognition, celebrity recognition, scene recognition, landmark recognition, artwork recognition, and optical character recognition (OCR). The accuracy rate of a single image for a single question and the full correct rate of a single image for two questions are used as perception indicators to calculate the score. 2.1) Evaluation of the perception capabilities of a large multimodal, multi-agent model. The evaluation primarily assesses ten sub-tasks across three main categories: coarse-grained recognition, fine-grained recognition, and OCR.

[0009] 2.2) Calculate the scores for each sub-task of the perceptual ability assessment. The ten sub-tasks of the perceptual ability assessment include one correct answer to one instruction for each image and two correct answers to two instructions for each image. Participants answer "yes" or "no" to the questions and then compare their answers with the standard answers. Calculate the score for each sub-task. The calculation formula is as follows:

[0010] in, Indicates the first stage of the perception task The score for each subtask is calculated by the response rate for each instruction corresponding to each image in each subtask. Response rate for each subtask corresponding to two instructions per image Add them together and multiply by 100 to get the result. and The range is 0-1, so multiplying by 100 makes it easier to observe.

[0011] 2.3) Based on the scores of the ten sub-tasks, the total score of the multimodal multi-agent large model's perception capability can be calculated, which reflects the overall perception capability of the multi-agent model. The calculation formula is as follows:

[0012] in, This represents the total score across ten perception tasks, which is the overall score for perception ability. A higher value indicates better perception ability of the multimodal, multi-agent large model, and stronger image perception ability. It can be used to test the perception and recognition ability of a multimodal, large model, with a maximum score of 2000.

[0013] Step 3) Multimodal multi-agent cognitive ability assessment, including four sub-tasks of reasoning, including common sense reasoning, numerical calculation, text translation and code reasoning. The accuracy rate of single picture single question and the full correct rate of single picture double question are used as cognitive indicators to calculate the score. The basic score of MME is output based on the perception score in Step 2). 3.1) Cognitive ability assessment using a multimodal, multi-agent large model. The assessment primarily evaluates four sub-tasks: commonsense reasoning, numerical computation, text translation, and code reasoning.

[0014] 3.2) Calculate the scores for each sub-task of cognitive ability. The four sub-tasks of the cognitive ability assessment include one correct answer to one instruction and two correct answers to two instructions per image. Participants answer "yes" or "no" to the questions and then compare their answers with the standard answers. Calculate the score for each sub-task. The calculation formula is as follows:

[0015] in, This indicates the cognitive task number 1 The score for each subtask is calculated by the response rate for each instruction corresponding to each image in each subtask. Response rate for each subtask corresponding to two instructions per image Add them together and multiply by 100 to get the result. and The range is 0-1, so multiplying by 100 makes it easier to observe.

[0016] 3.3) Based on the scores of the four sub-tasks, the total score of the multimodal multi-agent large model's cognitive ability can be calculated, which reflects the overall cognitive ability of the multi-agent. The calculation formula is as follows:

[0017] in, This represents the total score across four cognitive tasks, which is the overall cognitive ability score. A higher value indicates better cognitive ability and stronger textual reasoning ability in the multimodal, multi-agent model. It can be used to test the cognitive reasoning ability of a multimodal, multi-agent model, with a maximum score of 800.

[0018] 3.4) Based on the perception ability score and the cognitive ability score, calculate the comprehensive ability score of the multimodal multi-agent large model for image and text perception, recognition and reasoning. The calculation formula is as follows:

[0019] in, This represents the overall capability score of a multimodal, multi-agent large model. The larger the value, the stronger the model's comprehensive ability to perceive, understand, and reason about multimodal inputs, providing a fair and comprehensive standard for the performance measurement and optimization of MLLM.

[0020] Step 4) Extract target keywords from the generated text or user reference instructions of the multimodal agent, obtain the actual object category set by detecting the image to be evaluated through YOLOv8, and match the keywords with the object categories by combining the preset Chinese and English object mapping table, and calculate the image element matching score. 4.1) Construct a Chinese-English object mapping table. This is achieved through three main steps: defining forward mapping, constructing reverse mapping, and multi-scene matching and adaptation. Each Chinese keyword corresponds to one or more English categories, ensuring that both Chinese and English keywords can be accurately associated with the English object categories output by YOLOv8.

[0021] 4.2) Implementation of extracting target keywords from user reference instructions. The input text, including the generated text or reference instructions, is cleaned, including removing punctuation, extra spaces, and ensuring consistent lowercase for both Chinese and English. Word segmentation and part-of-speech tagging are performed on the processed text. The Chinese word segmentation tool jieba is used for Chinese text, and the English word segmentation tool NLTK is used for English text. A candidate keyword dictionary is constructed, retaining only words in the text that belong to this dictionary, ensuring that the extracted keywords match the object categories detectable by YOLOv8.

[0022] 4.3) Load the YOLOv8 object detection model to recognize the images generated by the multimodal multi-agent model, obtain the object category index, convert the index to lowercase English object category, and obtain the actual object category set after deduplication.

[0023] The Yolov8 algorithm directly employs a dynamic positive and negative sample allocation strategy. Positive samples are selected based on a weighted score calculated from the classification and regression scores. The formula is as follows:

[0024] in, It is the predicted score corresponding to the labeled category. It is the intersection-union ratio of the predicted bounding box and the ground truth bounding box. Multiplying the two together measures the degree of alignment. and It is a weight hyperparameter. The alignment can be measured by simultaneously controlling the optimization of classification scores and intersection-over-union ratio (IoU). Higher category scores and higher IoU ratios result in better alignment. The closer the value is to 1.

[0025] The Yolov8 algorithm's loss calculation consists of two branches: classification and regression. The classification branch uses BCE Loss, calculated using the following formula:

[0026] in, It is a binary tag 0 or 1. It outputs the probability of belonging to the label. This indicates the number of groups of objects predicted by the model. The binary cross-entropy loss function is used to evaluate the quality of a binary classification model's prediction results.

[0027] The regression branch uses the DFL (Distribution Focal Loss) loss function formula, which is calculated as follows:

[0028] in, It is the probability of the true distribution. It is the probability of the predicted distribution.

[0029] 4.4) Matching based on identified elements. The system matches keywords with object categories using a pre-defined Chinese-English object mapping table. If the keyword is in Chinese, it matches the corresponding English category from the mapping table and checks if it exists in the actual object category set. If the keyword is in English, it directly checks if it exists in the actual object category set, or it performs a reverse match of the corresponding Chinese category from the mapping table and then checks the result, counting the number of matched elements.

[0030] 4.5) Calculate the accuracy of image element generation based on the number of matched elements and the number of extracted target keywords. The calculation formula is as follows:

[0031] in, This indicates the number of matched elements. This indicates the number of target keywords extracted. This score represents the model's image generation capability. A higher score indicates that the generated image contains more elements and the model has a stronger image generation capability.

[0032] Step 5) Preset the mapping relationship between function types and API call vectors. The API call vector is a three-dimensional vector, which corresponds to the text generation, image generation and image description modules respectively. Extract the API call history of the agent and filter out the set of calling modules to generate the actual API call vector. Calculate the matching degree between the actual vector and the preset vector and output the API call matching degree score. 5.1) Construct Function-Vector Mapping Preset. Establish a one-to-one correspondence between function types and 3D API call vectors. The three dimensions of the vector correspond to the text generation, image generation, and image description modules, respectively. The specific mapping relationship is as follows: Text generation function corresponding vector ; Image generation function corresponding vector ; Image description function corresponding vector ; Image generation and image description function correspondence vectors ; The formula for constructing a function-vector mapping is as follows:

[0033] in, This indicates that the text calls the function module. This indicates the image generation function module. This represents the image description function module. A dimension value of 1 indicates that the multi-agent system called this module, while a dimension value of 0 indicates that the multi-agent system did not call this module. This indicates the status of multiple agents executing a single API call.

[0034] 5.2) Actual call vector generation. Iterate through each call record in the list of stored API call records, extract the module call records, remove duplicates to obtain the actual call module set. If the module exists in the module name corresponding to the vector dimension, set the dimension value to 1; otherwise, set it to 0, thus generating the actual API call vector.

[0035] 5.3) API Interface Matching Score Calculation. Based on the user's input requirements, select the functional module to be called by the multi-agent system in this instance. Find the correct corresponding vector that will be used in this call from the preset vectors. Count the number of times the actual call vector has the same dimension value as the preset vector. If the corresponding dimensions are the same, it is the matching dimension count. The formula for calculating the API call matching score is as follows:

[0036] in, This indicates the number of dimensions in the match. This represents the API interface matching score, which is the score for correctly calling different module APIs. The higher the score, the more API interfaces were correctly invoked, indicating stronger internal collaboration capabilities within the multi-agent system.

[0037] Step 6) Based on Steps 3), 4), and 5), integrate the MME base score, image element matching score, and API call matching score to output a comprehensive evaluation result.

[0038] 6.1) Normalize the perceptual and cognitive scores obtained in steps 3) and 4). Total Perception Score The total cognitive score is normalized to 0-10 points, and the calculation formula is as follows:

[0039] in, This represents the normalized score for multimodal multi-agent systems, with a score range of 0-10. 6.2) Calculate the multi-dimensional evaluation score of the large multimodal, multi-agent model that includes an image generation module. If the evaluation object includes an image modality, i.e., includes an image generation module, calculate the multi-dimensional score of the model. The calculation formula is as follows:

[0040] in, This is the model's multi-dimensional score, and the score calculation formula is used assuming the model includes image generation functionality. The higher the score, the better the overall performance of the multimodal multi-agent large model after the fusion of various dimensions. 6.3) Calculate the multi-dimensional evaluation score of a large multimodal, multi-agent model that does not include an image generation module. If the evaluation object includes an image modality (i.e., excluding the image generation module and only including text output, image recognition, and description modules), calculate the multi-dimensional score of the model. The calculation formula is as follows:

[0041] in, This is the model's multi-dimensional score, and the score calculation formula is used when the model does not include image generation functionality. The higher the score, the better the overall performance of the multimodal multi-agent large model after the fusion of various dimensions. 6.4) Multi-dimensional evaluation of the performance of a multimodal, multi-agent large-scale model. The final score is obtained by rounding down to one decimal place through multi-dimensional scoring. The final score evaluation criteria are as follows:

[0042] in, This is the final performance evaluation of the model, with a score range of 0-10. or Scores are given, with 8.0-10.0 being excellent, 6.0-7.9 being good, and 0-5.9 being areas for improvement.

[0043] The beneficial effects of inventions and creations are: This invention combines the 14 sub-tasks of MME, fine-grained object detection, and functional evaluation via API calls to achieve multi-dimensional coverage of the core capabilities of multimodal agents. It ensures comprehensive evaluation by considering MME standards, improves image evaluation accuracy through object detection algorithms, and fills the gaps in functional execution quantification through API call evaluation. The synergy of these three elements significantly enhances evaluation reliability and makes it highly practical. Both the Chinese and English object mapping tables and API function vectors can be dynamically expanded, flexibly adapting to new modalities such as video generation and industry-specific object recognition scenarios, demonstrating strong scalability. Independent output of scores for each dimension facilitates rapid identification of deficiencies such as insufficient agent perception capabilities and API call errors. This invention effectively solves the problem of modal fragmentation in existing evaluations, requires no manual intervention throughout the entire process, supports batch evaluation, has a high degree of automation, and is adaptable to large-scale agent performance result detection scenarios. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0045] like Figure 1 As shown, the steps of the multi-agent perception and cognition generation and API call multimodal evaluation method are as follows: Step 1) Using a publicly available standardized dataset, download the MME dataset and deploy it locally. Input the image generation command, which involves multiple objects. 1.1) Download the open-source MME dataset. This dataset comprehensively evaluates the perceptual and cognitive abilities of MLLM, with the two abilities supporting each other. The dataset is diverse; some images are sampled from public datasets, but only the images themselves are used, without relying on the original annotations. It is supplemented with a large number of manually taken images, including handwritten arithmetic problems, photos of everyday scenes, and images generated by the diffusion model, further reducing the risk of leakage.

[0046] 1.2) Load the perceptual ability dataset. This dataset is mainly divided into three categories: coarse-grained recognition, fine-grained recognition, and OCR. Coarse-grained recognition is used to evaluate the recognition of basic attributes of common objects, including four sub-tasks; fine-grained recognition is used to evaluate the professional recognition of content in specific domains, including five sub-tasks; the OCR task is used to evaluate image-to-text recognition capabilities, selecting only simple text samples, containing 20 images and 40 sets of instruction-answer pairs.

[0047] 1.3) Load the cognitive ability dataset. This mainly includes four sub-tasks: commonsense reasoning, numerical computation, text translation, and code reasoning. All images for the cognitive tasks were taken manually, and the instruction-answer pairs were designed manually to ensure no data leakage.

[0048] 1.4) Obtain image generation instructions. Input the image generation instructions, which mainly include generating multiple elements within an image. Load the object detection model to support fast detection of object categories in the image.

[0049] Step 2) Perform multi-dimensional capability assessment based on MME standard to evaluate the perception capabilities of multiple agents, including ten perception sub-tasks, mainly including coarse-grained recognition, fine-grained recognition, and OCR, and use the single-image single-question accuracy rate and the single-image double-question full correct rate as perception indicators to calculate the score. 2.1) Evaluation of the perception capabilities of a large multimodal, multi-agent model. The evaluation primarily assesses ten sub-tasks across three main categories: coarse-grained recognition, fine-grained recognition, and OCR.

[0050] 2.2) Calculate the scores for each sub-task of the perceptual ability assessment. The ten sub-tasks of the perceptual ability assessment include one correct answer to one instruction for each image and two correct answers to two instructions for each image. Participants answer "yes" or "no" to the questions and then compare their answers with the standard answers. Calculate the score for each sub-task. The calculation formula is as follows:

[0051] in, Indicates the first stage of the perception task The score for each subtask is calculated by the response rate for each instruction corresponding to each image in each subtask. Response rate for each subtask corresponding to two instructions per image Add them together and multiply by 100 to get the result. and The range is 0-1, so multiplying by 100 makes it easier to observe.

[0052] 2.3) Based on the scores of the ten sub-tasks, the total score of the multimodal multi-agent large model's perception capability can be calculated, which reflects the overall perception capability of the multi-agent model. The calculation formula is as follows:

[0053] in, This represents the total score across ten perception tasks, which is the overall score for perception ability. A higher value indicates better perception ability of the multimodal, multi-agent large model, and stronger image perception ability. It can be used to test the perception and recognition ability of a multimodal, large model, with a maximum score of 2000.

[0054] Step 3) Multimodal multi-agent cognitive ability assessment, including four sub-tasks of reasoning, including common sense reasoning, numerical calculation, text translation and code reasoning. The accuracy rate of single picture single question and the full correct rate of single picture double question are used as cognitive indicators to calculate the score. The basic score of MME is output based on the perception score in Step 2). 3.1) Cognitive ability assessment using a multimodal, multi-agent large model. The assessment primarily evaluates four sub-tasks: commonsense reasoning, numerical computation, text translation, and code reasoning.

[0055] 3.2) Calculate the scores for each sub-task of cognitive ability. The four sub-tasks of the cognitive ability assessment include one correct answer to one instruction and two correct answers to two instructions per image. Participants answer "yes" or "no" to the questions and then compare their answers with the standard answers. Calculate the score for each sub-task. The calculation formula is as follows:

[0056] in, This indicates the cognitive task number 1 The score for each subtask is calculated by the response rate for each instruction corresponding to each image in each subtask. Response rate for each subtask corresponding to two instructions per image Add them together and multiply by 100 to get the result. and The range is 0-1, so multiplying by 100 makes it easier to observe.

[0057] 3.3) Based on the scores of the four sub-tasks, the total score of the multimodal multi-agent large model's cognitive ability can be calculated, which reflects the overall cognitive ability of the multi-agent. The calculation formula is as follows:

[0058] in, This represents the total score across four cognitive tasks, which is the overall cognitive ability score. A higher value indicates better cognitive ability and stronger textual reasoning ability in the multimodal, multi-agent model. It can be used to test the cognitive reasoning ability of a multimodal, multi-agent model, with a maximum score of 800.

[0059] 3.4) Based on the perception ability score and the cognitive ability score, calculate the comprehensive ability score of the multimodal multi-agent large model for image and text perception, recognition and reasoning. The calculation formula is as follows:

[0060] in, This represents the overall capability score of a multimodal, multi-agent large model. The larger the value, the stronger the model's comprehensive ability to perceive, understand, and reason about multimodal inputs, providing a fair and comprehensive standard for the performance measurement and optimization of MLLM.

[0061] Step 4) Extract target keywords from the generated text or user reference instructions of the multimodal agent, obtain the actual object category set by detecting the image to be evaluated through YOLOv8, and match the keywords with the object categories by combining the preset Chinese and English object mapping table, and calculate the image element matching score. 4.1) Construct a Chinese-English object mapping table. This is achieved through three main steps: defining forward mapping, constructing reverse mapping, and multi-scene matching and adaptation. Each Chinese keyword corresponds to one or more English categories, ensuring that both Chinese and English keywords can be accurately associated with the English object categories output by YOLOv8.

[0062] 4.2) Implementation of extracting target keywords from user reference instructions. The input text, including the generated text or reference instructions, is cleaned, including removing punctuation, extra spaces, and ensuring consistent lowercase for both Chinese and English. Word segmentation and part-of-speech tagging are performed on the processed text. The Chinese word segmentation tool jieba is used for Chinese text, and the English word segmentation tool NLTK is used for English text. A candidate keyword dictionary is constructed, retaining only words in the text that belong to this dictionary, ensuring that the extracted keywords match the object categories detectable by YOLOv8.

[0063] 4.3) Load the YOLOv8 object detection model to recognize the images generated by the multimodal multi-agent model, obtain the object category index, convert the index to lowercase English object category, and obtain the actual object category set after deduplication.

[0064] The Yolov8 algorithm directly employs a dynamic positive and negative sample allocation strategy. Positive samples are selected based on a weighted score calculated from the classification and regression scores. The formula is as follows:

[0065] in, It is the predicted score corresponding to the labeled category. It is the intersection-union ratio of the predicted bounding box and the ground truth bounding box. Multiplying the two together measures the degree of alignment. and It is a weight hyperparameter. The alignment can be measured by simultaneously controlling the optimization of classification scores and intersection-over-union ratio (IoU). Higher category scores and higher IoU ratios result in better alignment. The closer the value is to 1.

[0066] The Yolov8 algorithm's loss calculation consists of two branches: classification and regression. The classification branch uses BCE Loss, calculated using the following formula:

[0067] in, It is a binary tag 0 or 1. It outputs the probability of belonging to the label. This indicates the number of groups of objects predicted by the model. The binary cross-entropy loss function is used to evaluate the quality of a binary classification model's prediction results.

[0068] The regression branch uses the DFL (Distribution Focal Loss) loss function formula, which is calculated as follows:

[0069] in, It is the probability of the true distribution. It is the probability of the predicted distribution.

[0070] 4.4) Matching based on identified elements. The system matches keywords with object categories using a pre-defined Chinese-English object mapping table. If the keyword is in Chinese, it matches the corresponding English category from the mapping table and checks if it exists in the actual object category set. If the keyword is in English, it directly checks if it exists in the actual object category set, or it performs a reverse match of the corresponding Chinese category from the mapping table and then checks the result, counting the number of matched elements.

[0071] 4.5) Calculate the accuracy of image element generation based on the number of matched elements and the number of extracted target keywords. The calculation formula is as follows:

[0072] in, This indicates the number of matched elements. This indicates the number of target keywords extracted. This score represents the model's image generation capability. A higher score indicates that the generated image contains more elements and the model has a stronger image generation capability.

[0073] Step 5) Preset the mapping relationship between function types and API call vectors. The API call vector is a three-dimensional vector, which corresponds to the text generation, image generation and image description modules respectively. Extract the API call history of the agent and filter out the set of calling modules to generate the actual API call vector. Calculate the matching degree between the actual vector and the preset vector and output the API call matching degree score. 5.1) Construct Function-Vector Mapping Preset. Establish a one-to-one correspondence between function types and 3D API call vectors. The three dimensions of the vector correspond to the text generation, image generation, and image description modules, respectively. The specific mapping relationship is as follows: Text generation function corresponding vector ; Image generation function corresponding vector ; Image description function corresponding vector ; Image generation and image description function correspondence vectors ; The formula for constructing a function-vector mapping is as follows:

[0074] in, This indicates that the text calls the function module. This indicates the image generation function module. This represents the image description function module. A dimension value of 1 indicates that the multi-agent system called this module, while a dimension value of 0 indicates that the multi-agent system did not call this module. This indicates the status of multiple agents executing a single API call.

[0075] 5.2) Actual call vector generation. Iterate through each call record in the list of stored API call records, extract the module call records, remove duplicates to obtain the actual call module set. If the module exists in the module name corresponding to the vector dimension, set the dimension value to 1; otherwise, set it to 0, thus generating the actual API call vector.

[0076] 5.3) API Interface Matching Score Calculation. Based on the user's input requirements, select the functional module to be called by the multi-agent system in this instance. Find the correct corresponding vector that will be used in this call from the preset vectors. Count the number of times the actual call vector has the same dimension value as the preset vector. If the corresponding dimensions are the same, it is the matching dimension count. The formula for calculating the API call matching score is as follows:

[0077] in, This indicates the number of dimensions in the match. This represents the API interface matching score, which is the score for correctly calling different module APIs. The higher the score, the more API interfaces were correctly invoked, indicating stronger internal collaboration capabilities within the multi-agent system.

[0078] Step 6) Based on Steps 3), 4), and 5), integrate the MME base score, image element matching score, and API call matching score to output a comprehensive evaluation result.

[0079] 6.1) Normalize the perceptual and cognitive scores obtained in steps 3) and 4). Total Perception Score The total cognitive score is normalized to 0-10 points, and the calculation formula is as follows:

[0080] in, This represents the normalized score for multimodal multi-agent systems, with a score range of 0-10. 6.2) Calculate the multi-dimensional evaluation score of the large multimodal, multi-agent model that includes an image generation module. If the evaluation object includes an image modality, i.e., includes an image generation module, calculate the multi-dimensional score of the model. The calculation formula is as follows:

[0081] in, This is the model's multi-dimensional score, and the score calculation formula is used assuming the model includes image generation functionality. The higher the score, the better the overall performance of the multimodal multi-agent large model after the fusion of various dimensions. 6.3) Calculate the multi-dimensional evaluation score of a large multimodal, multi-agent model that does not include an image generation module. If the evaluation object includes an image modality (i.e., excluding the image generation module and only including text output, image recognition, and description modules), calculate the multi-dimensional score of the model. The calculation formula is as follows:

[0082] in, This is the model's multi-dimensional score, and the score calculation formula is used when the model does not include image generation functionality. The higher the score, the better the overall performance of the multimodal multi-agent large model after the fusion of various dimensions. 6.4) Multi-dimensional evaluation of the performance of a multimodal, multi-agent large-scale model. The final score is obtained by rounding down to one decimal place through multi-dimensional scoring. The final score evaluation criteria are as follows:

[0083] in, This is the final performance evaluation of the model, based on... or Scores are given, with 8.0-10.0 being excellent, 6.0-7.9 being good, and 0-5.9 being areas for improvement.

[0084] Example 1: Given the loaded MME dataset and image generation instructions, evaluate the performance of a large multimodal, multi-agent model.

[0085] Step 1: Using a publicly available standardized dataset, download the MME dataset and deploy it locally.

[0086] Step 2: Test the perception capabilities of the large multi-agent model. Load the perception capability dataset. It mainly includes ten sub-tasks in three categories: coarse-grained recognition, fine-grained recognition, and OCR, and the perception capability score is MME_P: 1439.8.

[0087] Step 3: Test the cognitive function of the multi-agent large model. Load the cognitive ability dataset. Evaluate four sub-tasks: commonsense reasoning, numerical computation, text translation, and code reasoning, obtaining a cognitive function score MME_C: 590.3. Combining this with the MME_P obtained in Step 1, the MME_SUM is 2030.1.

[0088] Step 4: Input the image generation command to generate an example containing multiple elements. For example, generate an image of a kitten and a puppy playing on a playground. First, three predetermined targets are detected: cat, dog, and playground. The generated image then detects the cat and dog elements. The score is 6.7.

[0089] Step 5: Detect the image generation and image description function from Step 3. The corresponding preset vector table is as follows: The resulting vector is . The score is 10.0.

[0090] Step Six: Total Perception Score The normalized total cognitive score yielded an MME_N of 7.3. The multi-dimensional evaluation score of the multi-agent large model is also presented. The score is 7.9, which corresponds to a final performance rating of "good" for the model.

Claims

1. A multi-agent perception and cognition generation and API call multimodal evaluation method, characterized in that, The steps are as follows: Step 1) Using a publicly available standardized dataset, download the MME dataset and deploy it locally, then input the image generation command; Step 2) Perform multi-dimensional capability assessment based on MME standard to evaluate the perception capabilities of multiple agents, including ten perception sub-tasks, including object existence recognition, object counting, object position recognition, color recognition, movie poster recognition, celebrity recognition, scene recognition, landmark recognition, artwork recognition, and optical character recognition (OCR). The perception index score is calculated using the single image single question accuracy rate and the single image double question full correct rate. Step 3) Multi-agent cognitive ability assessment, including four reasoning sub-tasks: common sense reasoning, numerical calculation, text translation, and code reasoning. The accuracy rate of single question per image and the full correct rate of double questions per image are used as cognitive indicators to calculate the score. The basic MME score is output based on the perception index score in Step 2). Step 4) Extract target keywords from the generated text or user reference instructions of the multimodal agent, obtain the actual object category set by detecting the image to be evaluated through YOLOv8, and match the keywords with the object categories by combining the preset Chinese and English object mapping table, and calculate the image element matching score. Step 5) Preset the mapping relationship between function types and API call vectors. The API call vector is a three-dimensional vector, which corresponds to the text generation, image generation and image description modules respectively. Extract the API call history of the agent and filter out the set of calling modules to generate the actual API call vector. Calculate the matching degree between the actual vector and the preset vector and output the API call matching degree score. Step 6) Based on Steps 3), 4), and 5), integrate the MME base score, image element matching score, and API call matching score to output a comprehensive evaluation result.

2. The multi-agent perception and cognition generation and API call multimodal evaluation method according to claim 1, characterized in that, In step 1), the specific method is as follows: 1.1) Download the open-source dataset MME dataset: The dataset comprehensively evaluates the perceptual and cognitive abilities of MLLM, with the two abilities supporting each other; the dataset is diverse, with some images sampled from public datasets, but only the images themselves are used, without relying on the original annotations, and a large number of manually taken images are added, including handwritten arithmetic problems, photos of daily scenes, and images generated by diffusion models, to reduce the risk of leakage; 1.2) Loading the perceptual ability dataset: It includes three main categories: coarse-grained recognition, fine-grained recognition, and OCR. Coarse-grained recognition is used to evaluate the recognition of basic attributes of common objects, including four sub-tasks. Fine-grained recognition is used to evaluate the professional recognition of content in specific domains, including five sub-tasks. The OCR task is used to evaluate image text recognition ability, selecting only simple text samples, including 20 images and 40 sets of instruction-answer pairs. 1.3) Load the cognitive ability dataset: It includes four sub-tasks: common sense reasoning, numerical calculation, text translation, and code reasoning. The images for all cognitive tasks were taken manually, and the instruction-answer pairs were designed manually. 1.4) Obtain image generation instructions: Input image generation instructions, which include generating multiple elements in an image; load the object detection model to support the detection of object categories in the image.

3. The multi-agent perception and cognition generation and API call multimodal evaluation method according to claim 1, characterized in that, In step 2), the specific method is as follows: 2.1) Evaluation of perception capabilities of a large multimodal, multi-agent model: Evaluation of ten sub-tasks across three major categories: coarse-grained recognition, fine-grained recognition, and OCR; 2.2) Calculating the scores for each sub-task of perceptual ability assessment: The ten sub-tasks of perceptual ability assessment include one correct answer to one instruction and two correct answers to two instructions for each image. Participants answer "yes" or "no" to the questions, and then compare their answers with the standard answers to calculate the score for each sub-task. The calculation formula is as follows: in, Indicates the first stage of the perception task The score for each subtask is calculated by the response rate for each instruction corresponding to each image in each subtask. Response rate for each subtask corresponding to two instructions per image Add them together and multiply by 100 to get the result. and The range is 0-1, multiplying by 100 makes it easier to observe; 2.3) Calculate the total score of the multimodal multi-agent large model's perception capability based on the scores of the ten sub-tasks. This score reflects the overall perception capability of the multi-agent model. The calculation formula is shown below: in, This represents the total score across ten perception tasks, which is the overall score for perception ability. A higher value indicates better perception ability of the multimodal, multi-agent large model and stronger image perception ability. It is used to test the perception and recognition ability of a multimodal large model, with a maximum score of 2000.

4. The multi-agent perception and cognition generation and API call multimodal evaluation method according to claim 1, characterized in that, In step 3), the specific method is as follows: 3.1) Evaluation of cognitive abilities using a multimodal, multi-agent large model: Evaluation of four sub-tasks: common sense reasoning, numerical computation, text translation, and code reasoning; 3.2) Calculate the scores of each sub-task of cognitive ability: The test of the four sub-tasks of cognitive ability assessment includes one correct answer to one instruction and two correct answers to two instructions for each image. Answer yes or no to the question and compare with the standard answer to calculate the score of each sub-task. The calculation formula is as follows: in, This indicates the cognitive task number 1 The score for each subtask is calculated by the response rate for each instruction corresponding to each image in each subtask. Response rate for each subtask corresponding to two instructions per image Add them together and multiply by 100 to get the result. and The range is 0-1, multiplying by 100 makes it easier to observe; 3.3) Based on the scores of the four sub-tasks, the total score of the multimodal multi-agent large model's cognitive ability can be calculated, which reflects the overall cognitive ability of the multi-agent. The calculation formula is as follows: in, The total score represents the total cognitive ability score across four cognitive tasks. A higher score indicates better cognitive ability and stronger textual reasoning ability in a multimodal multi-agent model. This test is used to assess the cognitive reasoning ability of a multimodal model and has a maximum score of 800. 3.4) Based on the perception ability score and the cognitive ability score, calculate the comprehensive ability score of the multimodal multi-agent large model for image and text perception, recognition and reasoning. The calculation formula is as follows: in, This represents the overall capability score of a multimodal, multi-agent large model. The larger the value, the stronger the model's comprehensive ability to perceive, understand, and reason about multimodal inputs.

5. The multi-agent perception and cognition generation and API call multimodal evaluation method according to claim 1, characterized in that, In step 4), the specific method is as follows: 4.1) Constructing a Chinese-English object mapping table: This is achieved through three main steps: defining forward mapping, constructing reverse mapping, and adapting to multiple scenarios. Each Chinese keyword corresponds to one or more English categories, ensuring that both Chinese and English keywords can be accurately associated with the English object categories output by YOLOv8. 4.2) Implementation of extracting target keywords from user reference instructions: The input text, including the generated text or reference instructions, is cleaned, including removing punctuation marks, extra spaces, and unifying lowercase for both Chinese and English. Based on the processed text, word segmentation and part-of-speech tagging are performed. The Chinese word segmentation tool jieba is used to process the Chinese text, and the English word segmentation tool NLTK is used to process the English text. A candidate keyword dictionary is constructed, and only words in the text that belong to the dictionary are retained to ensure that the extracted keywords match the object categories that YOLOv8 can detect. 4.3) Load the YOLOv8 object detection model to identify images generated by the multimodal multi-agent model, obtain object category indices, convert the indices to lowercase English object categories, and obtain the actual object category set after deduplication; The Yolov8 algorithm directly employs a dynamic positive and negative sample allocation strategy, selecting positive samples based on a weighted score calculated from the classification and regression scores; the formula is as follows: in, It is the predicted score corresponding to the labeled category. It is the intersection-union ratio of the predicted bounding box and the ground truth bounding box. Multiplying the two together measures the degree of alignment. and It is a weight hyperparameter. Simultaneously, the alignment degree is measured by optimizing the classification score and intersection-over-union ratio (IoU). A higher classification score and IoU result in a better alignment. The closer the value is to 1, the better; The loss calculation in the Yolov8 algorithm consists of two branches: classification and regression. The classification branch uses BCE Loss, and the calculation formula is as follows: in, It is a binary tag 0 or 1. It outputs the probability that the label belongs to the target label. This indicates the number of groups of objects predicted by the model. The binary cross-entropy loss function is used to evaluate the quality of prediction results of a binary classification model; The regression branch uses the DFL (Distribution Focal Loss) loss function formula, which is calculated as follows: in, It is the probability of the true distribution. It is the probability of predicting the distribution; 4.4) Matching based on identified elements; Match keywords with object categories using a preset Chinese-English object mapping table. If the keyword is in Chinese, match the corresponding English category from the mapping table and determine if it exists in the actual object category set. If the keyword is in English, directly determine if it exists in the actual object category set, or match the corresponding Chinese category in reverse through the mapping table and then determine the number of matched elements. 4.5) The accuracy of image element generation is calculated based on the number of matched elements and the number of extracted target keywords. The calculation formula is as follows: in, This indicates the number of matched elements. This indicates the number of target keywords extracted. This score represents the model's image generation capability. A higher score indicates that the generated image contains more elements and the model has a stronger image generation capability.

6. The multi-agent perception and cognition generation and API call multimodal evaluation method according to claim 1, characterized in that, In step 5), the specific method is as follows: 5.1) Constructing Function-Vector Mapping Presets: Constructing a one-to-one correspondence between function types and 3D API call vectors. The three dimensions of the vector correspond to the text generation, image generation, and image description modules, respectively. The specific mapping relationship is as follows: Text generation function corresponding vector ; Image generation function corresponding vector ; Image description function corresponding vector ; Image generation and image description function corresponding vectors ; The formula for constructing a function-vector mapping is as follows: in, This indicates that the text calls the function module. This indicates the image generation function module. The dimension value represents the image description function module. A dimension value of 1 indicates that the multi-agent agent called this module, while a dimension value of 0 indicates that the multi-agent agent did not call this module. This indicates the status of multiple agents executing a single API call; 5.2) Actual call vector generation: Iterate through each call record in the list of stored API call records, extract the module call records, remove duplicates to obtain the actual call module set. If the module exists in the module name corresponding to the vector dimension, set the value of that dimension to 1, otherwise set it to 0, and generate the actual API call vector. 5.3) API Interface Matching Score Calculation: Based on the user's input requirements, select the functional module that the multi-agent needs to call in this instance. Find the correct corresponding vector that will be used in this call from the preset vector. Count the number of times the actual call vector has the same dimension value as the preset vector. If the corresponding dimensions are the same, that is, the matching dimension number, the API call matching score calculation formula is as follows: in, Indicates the number of dimensions in the match. This represents the API interface matching score, which is the score for correctly calling different module APIs. The higher the score, the more API interfaces were correctly invoked, indicating stronger internal collaboration capabilities within the multi-agent system.

7. The multi-agent perception and cognition generation and API call multimodal evaluation method according to claim 1, characterized in that, In step 6), the specific method is as follows: 6.1) Normalize the perceptual and cognitive scores obtained in steps 3) and 4). Total Perception Score The total cognitive score is normalized to 0-10 points, and the calculation formula is as follows: in, This represents the normalized score for multimodal multi-agent systems, with a score range of 0-10. 6.2) Calculate the multi-dimensional evaluation score of the large multimodal, multi-agent model that includes an image generation module. If the evaluation object includes an image modality, i.e., includes an image generation module, calculate the multi-dimensional score of the model using the following formula: in, This is a multi-dimensional score for the model, and the score calculation formula is used under the premise that the model includes image generation functionality; The higher the score, the better the overall performance of the multimodal multi-agent large model after the fusion of various dimensions. 6.3) Calculate the multi-dimensional evaluation score of a large multimodal, multi-agent model that does not include an image generation module. If the evaluation object includes an image modality (i.e., excluding the image generation module and only including text output, image recognition, and description modules), calculate the multi-dimensional score of the model using the following formula: in, This is a multi-dimensional score for the model, and the score calculation formula is used under the premise that the model does not include image generation functionality. The higher the score, the better the overall performance of the multimodal multi-agent large model after the fusion of various dimensions. 6.4) Multi-dimensional evaluation of the performance of a multimodal, multi-agent large model: The final score is obtained by rounding down to one decimal place through multi-dimensional scoring. The final score evaluation criteria are as follows: in, This is the final performance evaluation of the model, with a score range of 0-10. or Scores are given, with 8.0-10.0 being excellent, 6.0-7.9 being good, and 0-5.9 being areas for improvement.

Images