A method to realize the ecological operation of MaaS platform based on large model evaluation technology

Abstract

The present invention realizes the method of ecological operation of MaaS platform based on big model evaluation technology, including: determining the HTTP service interface and functional requirements of the model to be evaluated, and preparing the data set related to the model function; configuring the HTTP service address and OpenAPI interface document of the model to be evaluated; using the big model tool to generate the request parameter template through the prompt word, and replacing the file parameters with the placeholder; using the JMeter tool to set up the test environment; running the JMeter stress test process; writing Groovy scripts or using big model tools to generate code; using the script to read the generated VOC format message and compare it with the original data set. The present invention not only improves the model quality of the MaaS platform from a technical level, but also provides all-round improvements in operation management and user experience, laying a solid foundation for the long-term development of the platform.

Description

Technical Field

[0001] The present invention relates to the technical field of large model application and model evaluation, and in particular to a method for realizing ecological operation of a MaaS platform based on large model evaluation technology. Background Art

[0002] In the current MaaS operating platform, model suppliers put models on the shelves of the mall, and consumers order and use models according to their own needs. However, this model has significant shortcomings in the existing technology, especially when the performance of the models put on the shelves by the model suppliers is low, it will bring a series of problems. First, the existing platform cannot effectively evaluate whether the performance of the model meets the listing requirements, resulting in some low-performance models entering the mall. These models may show problems such as slow response speed, inaccurate results, and unsmooth functions in actual use, which directly affects the user experience and satisfaction. For example, when using the model-based intelligent recommendation function, if the recommended results are inaccurate and irrelevant, users will have a negative evaluation of the service quality of the platform, which will in turn affect their trust in the platform.

[0003] In addition, low-performance models will lead to a waste of platform operating resources. The mall platform needs to invest a lot of server resources and network resources to support the operation of these models. However, low-performance models often require more computing resources to maintain basic operations, and even unreasonable resource allocation occurs during operation. This waste of resources not only increases the operating costs of the platform, but also limits the operating efficiency of high-performance models, further reducing the overall operating efficiency of the platform.

[0004] After experiencing a poorly performing model, users may share their negative experience through word of mouth or social media. Such negative feedback will damage the business reputation of the mall platform, reduce the trust of potential users in the platform, and thus affect the platform's user growth and market share. In a highly competitive market, if other platforms can provide models and services with better performance, users will tend to choose competitors, which will lead to a decline in the market competitiveness of existing platforms and user loss.

[0005] Existing technologies fail to provide an effective mechanism for comprehensive performance evaluation and quality control of models. This allows low-performance models to be easily put on the shelves, bringing long-term operational risks to the platform. The existence of the above problems directly restricts the ecological and sustainable development of MaaS platform operations. Summary of the invention

[0006] In order to overcome the shortcomings of the existing technology, the present invention proposes a method for realizing the ecological operation of the MaaS platform based on large model evaluation technology. By strictly evaluating the performance and effect of the model, only high-quality models are allowed to be put on the mall, ensuring that users can obtain a fast, accurate and smooth service experience during use, avoiding user dissatisfaction caused by poor model performance.

[0007] To achieve the above objectives, the present invention provides a method for realizing the ecological operation of MaaS platform based on large model evaluation technology, comprising the following steps:

[0008] Step 1: Determine the HTTP service interface and functional requirements of the model to be evaluated, such as input parameters, calling methods, and return formats, and prepare data sets related to the model functions, including original files and annotated data;

[0009] Step 2: Configure the HTTP service address and OpenAPI interface document of the model to be evaluated, and ensure that the interface supports standardized requests and responses, and can normally receive HTTP requests and return results;

[0010] Step 3: Use the big model tool to generate a request parameter template based on the prompt words, replace the file parameter with a placeholder (such as $file), and ensure that the request parameter structure complies with the requirements of the interface document;

[0011] Step 4: Use the JMeter tool to set up the test environment, including thread groups, HTTP requests, parameter configuration, and stress test data loop processing scripts to achieve automatic loading and replacement of test data;

[0012] Step 5: Run the JMeter stress test process, initiate batch requests to the model service through the configured parameters, and save the service return results as a JSON format file for subsequent processing;

[0013] Step 6: Write a Groovy script or use a large model tool to generate code to convert JSON format data into VOC format messages, and save the generated VOC messages to the specified directory;

[0014] Step 7: Use the script to read the generated VOC format message, compare it with the original data set, complete the message conversion and data preparation, and provide a basis for subsequent analysis and use.

[0015] Furthermore, step 1 is as follows:

[0016] Step 1.1: Confirm that the model service has been successfully deployed, can receive HTTP requests, and has the ability to respond;

[0017] Step 1.2: Collect basic information about the model service, including the service URL, calling method (such as POST or GET), supported input parameters, and output format;

[0018] Step 1.3: Check whether the model service meets the functional requirements of the scenario to be evaluated, for example, whether the face detection model can recognize and return the face position and size information;

[0019] Step 1.4: Use a simple HTTP request tool (such as Postman) to send a test request to verify that the service can respond normally and return the expected results.

[0020] Furthermore, step 2 is as follows:

[0021] Step 2.1: Determine the type of test data required for the model function. For example, a face detection model requires a test set containing face images.

[0022] Step 2.2: Collect or create a dataset of original files related to the model function, ensuring that it contains valid input samples required by the model function;

[0023] Step 2.3: Prepare the annotation file corresponding to the dataset in VOC format (including labels, positions, and sizes), and ensure that the annotation information is comparable with the results returned by the model;

[0024] Step 2.4: Verify that the dataset content meets the following requirements:

[0025] a) Whether the original file is relevant to the model function;

[0026] b) Whether the annotation file format is in the standard VOC format, and whether the label and numerical information are accurate;

[0027] Step 2.5: If there is no ready-made dataset, you can download open source datasets or manually annotate the original files using annotation tools (such as Labelme) to generate annotated data that meets your needs.

[0028] Furthermore, step 3 is as follows:

[0029] Step 3.1: Choose a suitable large model service (such as Doubao or Tongyi Qianwen) and decide whether to use online mode or private deployment;

[0030] Step 3.2: Verify whether the large model service supports the functions of generating code and configuring request parameter templates to ensure that it is compatible with subsequent tasks;

[0031] Step 3.3: Write prompt words to clearly describe the model interface requirements and parameter structure, focusing on parameter names, formats, default values, and usage rules of file parameter placeholders;

[0032] Step 3.4: Use prompt words to communicate with the big model, generate a request parameter template that meets the model interface requirements, and replace the file type parameters with placeholders (such as $file);

[0033] Step 3.5: Check and verify the generated parameter template to ensure that the parameter name, format, and placeholder are consistent with the model service interface document.

[0034] Furthermore, step 4 is as follows:

[0035] Step 4.1: Write the configuration plan for the HTTP request based on the request parameter template generated in step 3 to ensure that the model service can be called correctly;

[0036] Step 4.2: Set the requested service URL, request method (POST / GET), header information, and request body structure to ensure that they match the definition in the interface document;

[0037] Step 4.3: Set the placeholders in the test dataset (such as $file) as dynamic variables, which can be replaced with real file paths or Base64-encoded file contents later;

[0038] Step 4.4: Use Postman or other testing tools to send requests to verify that the configured HTTP request can successfully call the model service and return the expected response;

[0039] Step 4.5: Adjust or correct possible format or logic problems in the request configuration to ensure that all parameters and request content comply with the definition of the interface document and the actual requirements of the model service.

[0040] Furthermore, step 5 is as follows:

[0041] Step 5.1: Install and start a performance testing tool (such as JMeter) and create a test plan for model stress testing;

[0042] Step 5.2: Add a Thread Group and set parameters such as the number of concurrent requests, number of loops, and start time to ensure that the request behavior under different load scenarios is simulated;

[0043] Step 5.3: Add HTTP Request Sampler and copy the HTTP request configuration verified in step 4 to Sampler, including service URL, request method and parameter content;

[0044] Step 5.4: Write a Groovy script to dynamically read the test dataset file, convert the file content to Base64 format, and assign it to the placeholder in the request parameter (such as $file);

[0045] Step 5.5: Configure JSR223 PreProcessor and bind the Groovy script to HTTP RequestSampler to ensure that the test data file is dynamically loaded every time a request is sent;

[0046] Step 5.6: Configure performance monitoring indicators, including throughput, response time (Average / Median / 90th Percentile), and error rate (Error%);

[0047] Step 5.7: Start the test plan and observe the performance indicators output by the tool in real time to ensure that the model service can run normally and respond under different loads.

[0048] Furthermore, step 6 is as follows:

[0049] Step 6.1: Make sure the model API returns results in JSON format and contains required field information, such as object category, coordinate location, and confidence score;

[0050] Step 6.2: Use the big model service to generate Groovy code that can parse the JSON return value and convert it into a VOC-formatted annotated message;

[0051] Step 6.3: Write a Groovy script to loop through the JSON files output by the performance test and convert each file from JSON to VOC format.

[0052] Step 6.4: Save the converted VOC format message as an XML file with the same name as the original file, ensuring that the file structure complies with the VOC standard;

[0053] Step 6.5: Verify the content of the converted VOC file and check whether the generated object labels and positions are consistent with the JSON data to ensure the correctness of the format and content.

[0054] Furthermore, step 7 is as follows:

[0055] Step 7.1: Load the annotated VOC messages in the test dataset and the VOC messages generated by the model, and ensure that the file names of the two correspond one to one;

[0056] Step 7.2: Use a self-developed engine or tool to parse the two sets of VOC messages and extract the target object category and its bounding box information;

[0057] Step 7.3: Compare the labeled positions of the same objects in the two groups of VOC messages through an algorithm, and calculate the position matching degree and classification accuracy;

[0058] Step 7.4: Output the comparison result data and save it in the evaluation report for subsequent analysis;

[0059] Step 7.5: Ensure that the calculation methods of all indicators during the evaluation process are consistent with industry standards so that the results are of reference value.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. The present invention provides a method for realizing the ecological operation of the MaaS platform based on large model evaluation technology. By pre-verifying the model performance, efficient models with reasonable resource occupancy are screened out, reducing the waste of server and network resources by low-performance models, effectively reducing the platform operation cost, and improving the overall resource utilization efficiency.

[0062] 2. The present invention provides a method for realizing the ecological operation of the MaaS platform based on large model evaluation technology. By providing better models and services, more users are attracted to choose the platform, thereby occupying an advantageous position in the market competition with competitors and promoting user growth and market share.

[0063] 3. The present invention provides a method for realizing the ecological operation of the MaaS platform based on large model evaluation technology. By providing a model evaluation report and multi-dimensional indicator data, it helps users to have clear reference standards when purchasing models, thereby improving the efficiency and accuracy of users' model selection.

[0064] 4. The present invention provides a method for realizing the ecological operation of the MaaS platform based on large model evaluation technology. By establishing a standardized model evaluation and listing mechanism, the management process of the MaaS platform is improved, forming a closed loop of high-quality model screening, reasonable resource allocation and user experience optimization, and promoting the healthy and sustainable development of the platform operation ecology. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 It is a schematic diagram of the steps of the present invention;

[0067] Figure 2 It is a schematic diagram of performance indicators. DETAILED DESCRIPTION

[0068] The technical solution of the present invention will be more clearly and completely explained below through description of preferred embodiments of the present invention in combination with the accompanying drawings.

[0069] Terminology explanation:

[0070] 1.JMeter: A commonly used performance testing tool used to simulate different concurrent loads and evaluate service performance.

[0071] 2.Thread Group: A thread group in JMeter that represents the number of concurrent users and their test behavior.

[0072] 3.HTTP Request Sampler: A component of JMeter used to configure and send HTTP requests to the target service.

[0073] 4.JSR223 PreProcessor: A script component in JMeter used to execute custom scripts (such as Groovy) before sending HTTP requests.

[0074] 5.Groovy: A dynamic scripting language, often used in JMeter to write custom scripts to process test data and parameters.

[0075] 6.Base64: An encoding method used to convert binary data (such as image files) into text format for transmission via HTTP.

[0076] 7.Throughput: A performance indicator that represents the number of requests processed by the service per unit time.

[0077] 8.Average / Median / 90th Percentile Response Time: An indicator that measures the average response time, median response time, and 90th percentile fastest response time of the service during the test.

[0078] 9.Error%: The percentage of failed requests during performance testing.

[0079] 10.VOC Format: A commonly used annotation format in computer vision datasets, containing information such as target labels and bounding box coordinates.

[0080] 11.JSON: JavaScript Object Notation, a lightweight data format commonly used in API responses.

[0081] 12. XML: Extensible Markup Language, used here to represent annotation data in VOC format.

[0082] like Figure 1As shown, the present invention is specifically:

[0083] The present invention proposes a method for realizing the ecological operation of the MaaS platform through large model evaluation technology. Before the supplier puts the product on the shelf, a multi-dimensional indicator evaluation report is output through model evaluation to verify whether the various indicators of the model performance meet the requirements for listing. Consumers can view the model evaluation report when ordering, which provides reference standards and expected management for the purchased model. In order to facilitate the use of operation and maintenance personnel without code development experience, the text generation technology of the large model is adopted to quickly generate the corresponding code through prompt words.

[0084] Preliminary preparation:

[0085] 1: Model to be evaluated. The model service has been deployed and supports normal HTTP calls. For example, for the face recognition model, the HTTP request parameter is an image file, and the upper left corner position, width and height of the face in the image are returned.

[0086] 2: Test data set. Select the corresponding data set (original file + annotated data) according to the model. The following two requirements must be met:

[0087] a) The relevance between the original file and the model. For example, if the evaluation model function is to recognize faces in pictures, it is necessary to select a picture dataset containing faces in the original file for effective evaluation;

[0088] b) Requirements for the annotation data format and the relevance of the annotation data model. The annotation format of the dataset is VOC. Relevance description: If the evaluation model function is to recognize faces in images, the annotation data needs to include face labels, face position, width and height information. The dataset can be obtained by downloading an open source dataset with face annotations, or by labelme annotation.

[0089] 3: Large model service. It can be privately deployed or completed using online services.

[0090] Model evaluation includes the following steps:

[0091] 1: Model service API configuration. Perform corresponding API configuration according to the HTTP service provided by the model to be evaluated. This includes setting the model's HTTP service address and openApi specification interface document. Please ensure that you can send HTTP requests to the request address and respond normally in the format defined in the interface document.

[0092] For example, take the online HTTP service of face detection.

[0093] 2: Talk to the big model and convert the request parameters. To adapt to different HTTP request parameters, you need to configure the mapping relationship. The value of the file in the parameter is filled with the placeholder $file so that it can be replaced with the real file later when JMeter sends the request. Combined with the interface document in step 1, the big model can be used to output the mapped request parameters, and the file is replaced with the placeholder $file (refer to the result example).

[0094] An example of a large model prompt word: the openapi is replaced with the interface document in step 1.

[0095] 3: Select the evaluation dataset. Select the appropriate VOC format dataset according to the model. In this example, the face detection dataset needs to be selected. The dataset should contain original messages and annotated messages.

[0096] 4: Configuration and operation of stress testing parameters.

[0097] This step integrates the JMeter stress testing tool. The specific configuration is as follows:

[0098] a) Add Thread Group and set parameters such as the number of concurrent requests, total number of requests, and request time;

[0099] b) Add HTTP Request and set the request parameters to the json obtained in step 2 ({ "image":"$file"});

[0100] c) Add JSR223 PreProcessor, set the groovy script to loop and read the evaluation data set, and set the parameter $file;

[0101] d) Set JMeter stress testing parameters, performance indicators, and post-processors.

[0102] Set JMeter stress test parameters, including: number of threads, startup time, number of loops, etc. Different stress test parameters will affect the stress test results. You need to perform multiple stress tests based on experience to obtain the best performance of the model.

[0103] Setting JMeter performance indicators includes: Throughput, Average, Median, 90th Percentile, Min, Max, Error%.

[0104] Set up a post-processor and use a regular expression extractor to save the request return results into a json text file.

[0105] e) Run JMeter and start stress testing.

[0106] The above steps can output model performance indicators, and subsequent steps can output model effect indicators.

[0107] 5: The large dialogue model generates groovy code and generates a VOC format annotation message. The format returned after the request is JSON, including the face position, width and height. The returned result needs to be converted into a VOC format message to compare with the dataset annotation data to obtain the model effect index. The large dialogue model can generate Python code.

[0108] 6: Add JSR223 PostProcessor, set the groovy script (generated in the previous step) to loop read the returned results and convert them into voc format files.

[0109] 7: Compare the output model effect indicators of the model annotation message and the data set annotation message. The performance indicators are as follows: Figure 2 shown.

[0110] The above implementation converts the model api return result into a voc format message, uses a self-developed engine to compare two voc messages, and calculates the following evaluation indicators. The comparison and calculation process is not convenient to be made public. The output industry common evaluation indicators include: Precision, Accuracy, Recall, F1-score, mAP, PR, ROC.

[0111] Through the above evaluation data, the MaaS platform can limit the indicator thresholds that models must meet before they can be put on the shelves, and refuse to put models that do not meet the thresholds on the shelves. Users can refer to the indicators of each dimension of the model when ordering to improve customer satisfaction.

[0112] The above steps mainly explain the overall process from the perspective of technical solutions.

[0113] As a specific implementation method, ensure that the model to be evaluated has been successfully deployed and can be called normally through the HTTP protocol. Get detailed information about the model service, including the service URL, request method (such as POST or GET), input parameters, and return format. Use tools such as Postman to verify whether the model service can return expected response data to ensure the normal operation of the interface function. Prepare a test data set according to the requirements of the model function, including raw data files and annotated data. It is recommended to use the VOC format for the annotated data format, and ensure that the annotated data is comparable to the results returned by the model. For example, the face detection model needs to use an annotated file containing the location of the face. If there is no ready-made data set, you can download it through an open source data set or generate it yourself by using an annotation tool such as Labelme.

[0114] Select a suitable big model service, such as Doubao or Tongyi Qianwen, which can be completed through private deployment or online service. Interact with the big model and generate code or request parameter templates that comply with the interface specifications through prompt words. Verify whether the generated template is accurate, including parameter name, type, default value, and whether the placeholders of file class parameters are set correctly. Prepare performance testing tools such as JMeter, create a test plan, add thread groups, and configure the number of concurrency, number of loops, and start time to simulate real load scenarios. Configure the HTTP request sampler, import the template generated by the big model, set the interface address, request method, and parameters. Write a Groovy script for the JSR223 preprocessor to dynamically read test data, encode the image file content into Base64, and fill it into the request parameters.

[0115] Run the performance test plan to monitor the performance of the service under different loads, and record key performance indicators such as throughput, average response time, and error rate. Receive JSON format data returned by the model service and extract the detected object category, coordinates, and confidence score. Use the code generated by the large model to convert the returned JSON results into a VOC format annotation file and save it as an XML file with the same name as the original file. Verify that the content of the generated VOC file is correct and ensure that its format is consistent with the annotation file of the test dataset. Compare the VOC file generated by the model with the annotation file in the test dataset and calculate performance indicators, including precision, recall, F1-score, and average precision.

[0116] Based on the evaluation results, determine whether the model meets the listing standards of the MaaS platform, and adjust or reject models that do not meet the requirements. Generate a model evaluation report to provide users with multi-dimensional reference data on model performance. Through the above process, a closed loop from model deployment, data preparation, performance testing to result evaluation is achieved to ensure the quality of model use on the platform and user satisfaction.

[0117] The above specific implementations are only descriptions of the preferred implementations of the present invention, and do not limit the protection scope of the present invention. Without departing from the design concept and spirit of the present invention, various modifications, substitutions and improvements made by ordinary technicians in this field to the technical solution of the present invention based on the text description and drawings provided by the present invention should all fall within the protection scope of the present invention. The protection scope of the present invention is determined by the claims.

Claims

1. A method for realizing the ecological operation of MaaS platform based on large model evaluation technology, characterized in that: The following steps are involved: Step 1: Determine the HTTP service interface and functional requirements of the model to be evaluated, and prepare data sets related to the model functions, including original files and annotated data; Step 2: Configure the HTTP service address and OpenAPI interface document of the model to be evaluated, ensure that the interface supports standardized requests and responses, and normally receives HTTP requests and returns results; Step 3: Use the big model tool to generate a request parameter template based on the prompt words, replace the file parameters with placeholders, and ensure that the request parameter structure meets the requirements of the interface document; Step 4: Use the JMeter tool to set up the test environment, including thread groups, HTTP requests, parameter configuration, and stress test data loop processing scripts to achieve automatic loading and replacement of test data; Step 5: Run the JMeter stress test process, initiate batch requests to the model service through the configured parameters, and save the service return results as a JSON format file for subsequent processing; Step 6: Write a Groovy script or use a large model tool to generate code to convert JSON format data into VOC format messages, and save the generated VOC messages to the specified directory; Step 7: Use the script to read the generated VOC format message, compare it with the original data set, and complete the message conversion and data preparation.

2. The method for realizing the ecological operation of MaaS platform based on large model evaluation technology according to claim 1 is characterized in that: Step 1 is as follows: Step 1.1: Confirm that the model service has been successfully deployed, can receive HTTP requests, and has the ability to respond; Step 1.2: Collect basic information about the model service, including the service URL, calling method, supported input parameters, and output format; Step 1.3: Check whether the model service meets the functional requirements of the scenario to be evaluated; Step 1.4: Use a simple HTTP request tool to send a test request to verify that the service can respond normally and return the expected results.

3. The method for realizing the ecological operation of MaaS platform based on large model evaluation technology according to claim 1 is characterized in that: Step 2 is as follows: Step 2.1: Determine the type of test data required for model functionality; Step 2.2: Collect or create a raw file dataset related to the model function, and ensure that the raw file dataset contains valid input samples required by the model function; Step 2.3: Prepare the annotation file corresponding to the dataset, using the VOC format, and ensure that the annotation information is comparable with the return results of the model; Step 2.4: Verify that the dataset content meets the following requirements: a) Whether the original file is relevant to the model function; b) Whether the annotation file format is in the standard VOC format, and whether the label and numerical information are accurate; Step 2.5: If there is no ready-made dataset, manually annotate the original file by downloading open source datasets or using annotation tools to generate annotated data that meets the requirements.

4. The method for realizing the ecological operation of MaaS platform based on large model evaluation technology according to claim 1 is characterized in that: Step 3 is as follows: Step 3.1: Select a large model service and decide whether to use online mode or private deployment; Step 3.2: Verify whether the large model service supports the functions of generating code and configuring request parameter templates to ensure compatibility with subsequent tasks; Step 3.3: Write prompt words to clearly describe the model interface requirements and parameter structure, focusing on parameter names, formats, default values, and usage rules of file parameter placeholders; Step 3.4: Use prompt words to communicate with the big model, generate a request parameter template that meets the model interface requirements, and replace the file class parameters with placeholders; Step 3.5: Check and verify the generated parameter template to ensure that the parameter name, format, and placeholder are consistent with the model service interface document.

5. The method for realizing the ecological operation of MaaS platform based on large model evaluation technology according to claim 1 is characterized in that: Step 4 is as follows: Step 4.1: Write the configuration plan for the HTTP request based on the request parameter template generated in step 3 to ensure that the model service can be called correctly; Step 4.2: Set the requested service URL, request method, header information, and request body structure to ensure that they match the definition in the interface document; Step 4.3: Set the placeholders in the test dataset as dynamic variables, which can be replaced with real file paths or Base64-encoded file contents later; Step 4.4: Use Postman or other testing tools to send requests to verify that the configured HTTP request can successfully call the model service and return the expected response; Step 4.5: Adjust or correct format or logic problems in the request configuration to ensure that all parameters and request content comply with the definition of the interface document and the actual requirements of the model service.

6. The method for realizing the ecological operation of MaaS platform based on large model evaluation technology according to claim 1 is characterized in that: Step 5 is as follows: Step 5.1: Install and start the performance testing tool, and create a new test plan for model stress testing; Step 5.2: Add a Thread Group and set parameters to ensure that the request behavior under different load scenarios is simulated; Step 5.3: Add HTTP Request Sampler and copy the HTTP request configuration verified in step 4 to Sampler, including service URL, request method and parameter content; Step 5.4: Write a Groovy script to dynamically read the test data set file, convert the file content into Base64 format, and assign it to the placeholder in the request parameter; Step 5.5: Configure JSR223 PreProcessor and bind the Groovy script to HTTP Request Sampler to ensure that the test data file is dynamically loaded every time a request is sent; Step 5.6: Configure performance monitoring indicators, including throughput, response time, and error rate; Step 5.7: Start the test plan and observe the performance indicators output by the tool in real time to ensure that the model service can run normally and respond under different loads.

7. The method for realizing the ecological operation of MaaS platform based on large model evaluation technology according to claim 1 is characterized in that: Step 6 is as follows: Step 6.1: Make sure the model API returns results in JSON format and contains the required field information; Step 6.2: Use the big model service to generate Groovy code. The Groovy code must be able to parse the JSON return value and convert it into a VOC formatted annotated message. Step 6.3: Write a Groovy script to loop through the JSON files output by the performance test and convert each file from JSON to VOC format. Step 6.4: Save the converted VOC format message as an XML file with the same name as the original file, ensuring that the file structure complies with the VOC standard; Step 6.5: Verify the converted VOC file content and check whether the generated object labels and positions are consistent with the JSON data.

8. The method for realizing the ecological operation of MaaS platform based on large model evaluation technology according to claim 1 is characterized in that: Step 7 is as follows: Step 7.1: Load the annotated VOC messages in the test dataset and the VOC messages generated by the model, and ensure that the file names of the two correspond one to one; Step 7.2: Use a self-developed engine or tool to parse the two sets of VOC messages and extract the target object category and its bounding box information; Step 7.3: Compare the labeled positions of the same objects in the two groups of VOC messages through an algorithm, and calculate the position matching degree and classification accuracy; Step 7.4: Output the comparison result data and save it in the evaluation report for subsequent analysis; Step 7.5: Ensure that the calculation methods of all indicators during the evaluation process are consistent with industry standards so that the results are of reference value.

Images