Quality infrastructure service efficiency evaluation method based on large model

Abstract

The invention discloses a quality infrastructure service efficiency evaluation method based on a large model, which is based on full-chain elements such as construction, management, service and propaganda and promotion of a quality infrastructure one-stop service platform, integrates application of a key technology of the one-stop service digital platform, deeply explores influence factors of online and offline service efficiency of the platform, and improves the quality infrastructure service efficiency evaluation efficiency. And a gradient grading platform evaluation system model is constructed. The objective of the invention is to provide a powerful tool for platform self-service efficiency evaluation and provincial platform gradient grading.

Description

Technical Field

[0001] The present invention relates to a quality infrastructure service effectiveness evaluation method, and more specifically to a quality infrastructure service effectiveness evaluation method based on a large model. Background Art

[0002] At present, the evaluation of the service effectiveness of quality infrastructure usually adopts traditional algorithm models such as AHP, TOPSIS model or fuzzy comprehensive evaluation method. However, these methods have many limitations, such as extremely high threshold requirements for data analysis, requiring extremely professional personnel to use complex mathematical tools for analysis and iteration, and strict requirements on data quality and integrity, and human intervention is required when necessary. Because of this, it is difficult to form a system that can evaluate continuously changing data in real time, and it is also difficult for non-professionals to participate in the operation. Summary of the invention

[0003] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a quality infrastructure service effectiveness evaluation method based on a large model, which saves manpower and material resources to a great extent and significantly improves the evaluation efficiency and accuracy.

[0004] To achieve the above object, the present invention provides the following technical solution: a quality infrastructure service performance evaluation method based on a large model, comprising the following steps: Step 1: Construct the five influencing factors of the evaluation and the overall system architecture; Step 2: Prepare a dataset for fine-tuning the large model; Step 3: fine-tune the large base model based on the data set prepared in step 2; Step 4: Evaluate the large model after fine-tuning different parameters in step 3; Step 5: Build service effectiveness evaluation service; Step 6: Perform service connection and output results; Step seven: Based on the output of step six, conduct result review and construct business modules related to institutional evaluation.

[0005] As a further improvement of the present invention, the specific method of constructing the five influencing factors of the evaluation and the overall architecture of the system in step one is: first, according to the characteristics of the "one-stop" service platform of the quality infrastructure, the five key influencing factors of the evaluation are established: service cycle, service price, user evaluation, customer service response efficiency, and data maintenance frequency; then, the overall architecture and network results of the system are designed.

[0006] As a further improvement of the present invention, the system constructed in step 1 includes: Access layer, used for communication connection with external PCs, mobile devices and other terminals; The WEB layer is used to provide PC applications, mobile applications and other applications for communication with PCs, mobile devices and other terminals; The interface layer is used to provide a communication interface; The service layer includes a service communication module, a business service module, a basic service component and a service management module, and is used to provide service communication, business service, basic service and service management functions; The resource layer includes the RDS database, OSS file storage, Redis cache, and platform site, which are used to provide computing resources for the system.

[0007] As a further improvement of the present invention, the specific method of fine-tuning the large base model in step three is: selecting a large model suitable for this application scenario as the basic model, using the LoRa scheme for fine-tuning, and training multiple candidate models by trying different LoRa parameter combinations.

[0008] As a further improvement of the present invention, the basic model selects glm4-9b-chat as the base large model, and takes four groups of parameters of r=8, alpha=16; r=8, alpha=32; r=16, alpha=16; r=16, alpha=32 for fine-tuning training.

[0009] As a further improvement of the present invention, the specific method of performing the evaluation in step 4 is: using a validation set to evaluate the fine-tuned model, so as to select the fine-tuned model with the best performance by comparing the performance of different models.

[0010] As a further improvement of the present invention, the specific method for evaluation in step 4 is: use a large model with different parameters fine-tuned to perform batch evaluation on the validation set respectively, and then use the variance of the difference between the model calculation results and the evaluation results in the validation set of the manual standard to calculate one by one, and select the model with the smallest difference.

[0011] Beneficial effects of the present invention: 1. Lowering the threshold for data analysis: The present invention uses a service performance evaluation method based on a large model, which does not rely on professional mathematical tools and human intervention, lowers the threshold for data analysis, and enables non-professionals to easily participate in the operation.

[0012] 2. Improve evaluation efficiency: Compared with traditional methods, this invention can evaluate service performance in real time, greatly improving evaluation efficiency. At the same time, through the self-learning and fine-tuning of the large model, the evaluation effect can be continuously optimized.

[0013] 3. Reduce manpower and material costs: Through automated evaluation and result review, the present invention can significantly reduce manpower and material costs and improve the economic benefits of evaluation work.

[0014] 4. Improve the accuracy of evaluation: The present invention uses a large model to evaluate service performance, which can more accurately capture the information in the data and improve the accuracy of the evaluation. At the same time, the reliability of the evaluation results can be further ensured by combining manual review and AI evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A network structure diagram of the quality infrastructure service effectiveness evaluation method based on a large model of the present invention; Figure 2 It is a flow chart of the quality infrastructure service effectiveness evaluation method based on a large model of the present invention; Figure 3 A system architecture diagram of the system constructed by the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0017] Reference Figures 1 to 3 As shown, a quality infrastructure service effectiveness evaluation method based on a large model in this embodiment includes the following steps: Step 1: Establish the five influencing factors of the evaluation and the overall system architecture.

[0018] First, based on the characteristics of the "one-stop" service platform of the quality infrastructure, five key influencing factors of the evaluation are established: service cycle, service price, user evaluation, customer service response efficiency, and data maintenance frequency. Then, the overall architecture and network results of the system are designed, as shown in the following figure. Figures 1 to 3 , among which Figure 3 The system architecture diagram shown is: MQ (Message Queue); RPC (Remote Procedure Call, remote procedure call); REST (Representational State Transfer); RDS (Relational Database Service); OSS (Object Storage Service, file storage service); Redis (Remote Dictionary Server, remote dictionary service).

[0019] Step 2: Prepare a dataset for fine-tuning a large model.

[0020] Construct a data set containing the above five influencing factors, and each sample has a corresponding service performance evaluation score. The data set needs to be strictly cleaned and labeled to ensure the accuracy and consistency of the data. The five influencing factors are standardized by column.

[0021] We invert the negative impact factors and then use the min-max scaling algorithm to normalize the data. Some sample codes are as follows: import numpy as np # Min-Max Scaling def min_max_scale(data): min_val = np.min(data) max_val = np.max(data) normalized_data = (data - min_val) / (max_val - min_val) return normalized_data scaled_data = min_max_scale(data).

[0022] Some examples of training data sets are as follows: {"messages": [{"role": "user", "content": "You are a service agency evaluation expert. Please give an evaluation based on the five influencing factors I gave, and only return excellent, good, qualified, and unqualified instead of other content; the five factors are: service cycle = 0.9, service price = 0.68, user evaluation = 0.86, customer service response efficiency = 0.93, data maintenance frequency = 0.97"}, {"role": "assistant", "content": "Excellent"}]}; {"messages": [{"role": "user", "content": "You are a service agency evaluation expert. Please give an evaluation based on the five influencing factors I gave, and only return excellent, good, qualified, and unqualified instead of other content; the five factors are: service cycle = 0.1, service price = 0.87, user evaluation = 0.74, customer service response efficiency = 0.62, data maintenance frequency = 0"}, {"role": "assistant", "content": "Qualified"}]}; {"messages": [{"role": "user", "content": "You are a service agency evaluation expert. Please give an evaluation based on the five influencing factors I gave, and only return excellent, good, qualified, and unqualified instead of other content; the five factors are: service cycle = 0.8, service price = 0.75, user evaluation = 0.94, customer service response efficiency = 0.76, data maintenance frequency = 0.9"}, {"role": "assistant", "content": "Excellent"}]}; {"messages": [{"role": "user", "content": "You are a service agency evaluation expert. Please give an evaluation based on the five influencing factors I gave, and only return excellent, good, qualified, and unqualified instead of other content; the five factors are: service cycle = 0.5, service price = 0.91, user evaluation = 0.81, customer service response efficiency = 0.28, data maintenance frequency = 0.53"}, {"role": "assistant", "content": "good"}]}; {"messages": [{"role": "user", "content": "You are a service agency evaluation expert. Please give an evaluation based on the five influencing factors I gave, and only return excellent, good, qualified, and unqualified instead of other content; the five factors are: service cycle = 0.75, service price = 0.71, user evaluation = 0.78, customer service response efficiency = 0.11, data maintenance frequency = 0.77"}, {"role": "assistant", "content": "good"}]}.

[0023] Step 3: Fine-tune the large base model.

[0024] Select a large model suitable for this application scenario as the basic model, and use the lora (Low-Rank Adaptation) solution for fine-tuning. By trying different lora parameter combinations, multiple candidate models are trained.

[0025] In this example, we choose glm4-9b-chat as the base model, and take four sets of parameters r=8,alpha=16; r=8,alpha=32; r=16,alpha=16; r=16,alpha=32 for fine-tuning training.

[0026] Step 4: Evaluate the large model after fine-tuning different parameters.

[0027] Use the validation set to evaluate the fine-tuned model. By comparing the performance of different models, select the fine-tuned model with the best performance.

[0028] The special evaluation method used in this example is to use a large model with different fine-tuned parameters to perform batch evaluation on the validation set, and then use the variance of the difference between the model calculation results and the manual standard validation set evaluation results to select the model with the smallest difference. In this example, the best effect is achieved when r=16, alpha=32, followed by r=8, alpha=16, but the latter has 36.7% lower resource overhead than the former.

[0029] Step 5: Build service effectiveness evaluation service.

[0030] Based on the fine-tuning model with the best performance, a service performance evaluation service is built. This service can receive the original influencing factor parameters from the "one-stop" service platform of the quality infrastructure, automatically standardize and call the large model interface for intelligent evaluation, and output the service performance evaluation results in real time.

[0031] Step 6: Service docking and result output.

[0032] Connect the service performance evaluation service with the "one-stop" service platform of the quality infrastructure to realize the automatic transmission of parameters and real-time output of results. Ensure that the service platform can obtain the service performance evaluation results in real time and optimize the service accordingly.

[0033] Step 7: Construction of business modules related to result review and institutional evaluation.

[0034] An audit module is added to the quality infrastructure "one-stop" service platform to manually review the service performance evaluation results output by the large model. Based on the final evaluation result data, the system builds a comprehensive evaluation display page for service agencies and an evaluation data statistical analysis module to provide data support for the service optimization of the quality infrastructure "one-stop" service platform.

[0035] In summary, the quality infrastructure service performance evaluation method based on the big model of this embodiment relies on artificial intelligence technology to innovate existing technologies. By using the manually cleaned service performance evaluation data set, the big model is fine-tuned many times, and an evaluation is carried out to select the best one. Then, an evaluation system is built based on the fine-tuned big model. This invention successfully resolves the problems existing in traditional data analysis methods, such as high thresholds, strict requirements on data quality, and time-consuming analysis, saving manpower and material resources to a great extent, while significantly improving evaluation efficiency and accuracy.

[0036] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A quality infrastructure service effectiveness evaluation method based on a large model, characterized by: The steps include: Step 1: Construct the five influencing factors of the evaluation and the overall system architecture; Step 2: Prepare a dataset for fine-tuning the large model; Step 3: fine-tune the large base model based on the data set prepared in step 2; Step 4: Evaluate the large model after fine-tuning different parameters in step 3; Step 5: Build service effectiveness evaluation service; Step 6: Perform service connection and output results; Step seven: Based on the output of step six, conduct result review and construct business modules related to institutional evaluation.

2. The method for evaluating the quality infrastructure service effectiveness based on a large model according to claim 1 is characterized by: The specific method of constructing the five influencing factors of the evaluation and the overall system architecture in the step one is: first, according to the characteristics of the "one-stop" service platform of the quality infrastructure, the five key influencing factors of the evaluation are established: service cycle, service price, user evaluation, customer service response efficiency, and data maintenance frequency. Then, the overall architecture and network results of the system are designed.

3. The method for evaluating the quality infrastructure service effectiveness based on a large model according to claim 2 is characterized by: The system constructed in step 1 includes: Access layer, used for communication connection with external PCs, mobile devices and other terminals; The WEB layer is used to provide PC applications, mobile applications and other applications for communication with PCs, mobile devices and other terminals; The interface layer is used to provide a communication interface; The service layer includes a service communication module, a business service module, a basic service component and a service management module, and is used to provide service communication, business service, basic service and service management functions; The resource layer includes the RDS database, OSS file storage, Redis cache, and platform site, which are used to provide computing resources for the system.

4. The method for evaluating the quality infrastructure service effectiveness based on a large model according to any one of claims 1 to 3, characterized in that: The specific method of fine-tuning the base large model in step three is: select a large model suitable for this application scenario as the basic model, use the LoRa solution for fine-tuning, and train multiple candidate models by trying different LoRa parameter combinations.

5. The method for evaluating the quality infrastructure service effectiveness based on a large model according to claim 4 is characterized by: The basic model selects glm4-9b-chat as the base large model, and takes four groups of parameters of r=8,alpha=16; r=8,alpha=32; r=16,alpha=16; r=16,alpha=32 for fine-tuning training.

6. The method for evaluating the quality infrastructure service effectiveness based on a large model according to any one of claims 1 to 3, characterized in that: The specific method of performing the evaluation in the step 4 is: using the validation set to evaluate the fine-tuned model, so as to select the fine-tuned model with the best performance by comparing the performance of different models.

7. The method for evaluating the quality infrastructure service effectiveness based on a large model according to claim 6 is characterized by: The specific method for evaluation in step 4 is: use the large model with different parameters fine-tuned to perform batch evaluation on the validation set respectively, and then use the variance of the difference between the model calculation results and the manual standard validation set evaluation results to calculate one by one, and select the model with the smallest difference.

Images