Operation effect evaluation method and device, equipment and storage medium

Abstract

This application discloses an operational effectiveness evaluation method, apparatus, device, and storage medium, relating to the field of information processing technology. The operational effectiveness evaluation method includes: acquiring a target scenario and evaluation time interval configured by a user; inputting the target scenario and the evaluation time interval into an operational effectiveness evaluation model to obtain an operational effectiveness evaluation summary output by the operational effectiveness evaluation model. The operational effectiveness evaluation model consists of a single base model and multiple scenario-specific adaptation modules, and is obtained through multi-scenario-specific fine-tuning training on scenario training data. This application, by setting a single base and multiple adaptation module architecture, avoids data interference between scenarios, accurately adapts to the specific needs of each scenario, and eliminates the need to deploy multiple independent models, significantly saving computational and storage resources for training and inference. Furthermore, multi-scenario-specific fine-tuning ensures the scenario adaptability and accuracy of the evaluation summary.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to a method, apparatus, equipment and storage medium for evaluating operational effectiveness. Background Technology

[0002] In the process of multi-scenario business operation, in order to evaluate the operational effectiveness of scenario products, optimize resource allocation and user experience, it is necessary to summarize the recommendation strategies and operational content performance of each scenario. The industry mainly adopts the traditional unified model prediction solution to carry out the scenario performance summary work. This type of solution attempts to cover the summary needs of multiple different scenarios with a single model, and follows a unified design idea in model construction and application.

[0003] However, this existing technology has significant drawbacks: different scenarios have significant differences in data distribution, business objectives and text style, and a unified model is prone to interference between scenarios, making it difficult to take into account the specificity of each scenario. Moreover, as the number of scenarios increases, the number of models will increase linearly, leading to a sharp increase in the consumption of computing and storage resources for training and inference. Summary of the Invention

[0004] The main purpose of this application is to provide an operational effectiveness evaluation method, apparatus, equipment, and storage medium, which aims to accurately adapt to the needs of each scenario, ensure the specificity of scenario summaries, and save resources.

[0005] To achieve the above objectives, this application proposes an operational effectiveness evaluation method, the method comprising: Obtain the target scenario and evaluation time range configured by the user; The target scenario and the evaluation time interval are input into the operation effect evaluation model to obtain the operation effect evaluation summary output by the operation effect evaluation model. The operation effect evaluation model consists of a single base model and multiple scenario-specific adaptation modules. The operation effect evaluation model is obtained by performing multi-scenario-specific fine-tuning training on scenario training data.

[0006] In one possible implementation, before inputting the target scenario and the evaluation time interval into the operational performance evaluation model to obtain the operational performance evaluation summary output by the operational performance evaluation model, the method further includes: Construct scene training data that is adaptable to multiple scenarios; Based on the training data of the aforementioned scenarios, the single base model is subjected to multi-scenario-specific fine-tuning training. The training process is optimized by integrating model quantization, gradient accumulation, and dynamic sparsity mechanism of model parameters, forming an operation effect evaluation model equipped with multiple scenario-specific adaptation modules. The operation effect evaluation model includes the candidate scenario expert models that have been trained for each scenario. When a new scenario is identified, a target scenario expert model that is related to the new scenario in terms of scenario attributes or business characteristics is selected from the candidate scenario expert models, and transfer learning is performed based on the target scenario expert model to complete the adaptation of the new scenario.

[0007] In one possible implementation, constructing scene training data adapted to multiple scenarios includes: Obtain the contextual information and historical user behavior data corresponding to each business scenario; A scene background knowledge base is constructed based on the scene background information, and the historical user behavior data is converted into vector form and stored to construct a behavior vector library. Obtain daily user behavior statistics for each of the aforementioned business scenarios, and retrieve relevant historical behavior data from the behavior vector library based on the daily user behavior statistics. By using retrieval-enhanced generation technology, the retrieved relevant historical behavior data and the corresponding scene background information in the scene background knowledge base are input into a general large model to generate scene training data adapted to each of the aforementioned business scenarios.

[0008] In one possible implementation, based on the scene training data, the single base model undergoes multi-scene-specific fine-tuning training, integrating model quantization, gradient accumulation, and dynamic sparsity mechanisms for model parameters to optimize the training process, forming an operational performance evaluation model equipped with multiple scene-specific adaptation modules, including: The single base model is subjected to parameter quantization processing; For any business scenario, based on the scenario training data corresponding to the business scenario, a scenario-specific adaptation module corresponding to the business scenario is trained on a specified projection layer of the single base model. During the training process, the training batch is split by gradient accumulation and a dynamic sparsity mechanism for model parameters is set to sparsify the weights with low contribution in the model and reactivate the sparsified weights that meet the activation conditions. After training of all scenario-specific adaptation modules is completed, each scenario-specific adaptation module is integrated with a single base model to form an operational performance evaluation model that includes candidate scenario expert models corresponding to each business scenario.

[0009] In one possible implementation, the dynamic sparsity mechanism for setting model parameters includes: In the initial training phase, the weight gradient of the specified projection layer is globally sampled, and the weights with gradient contributions lower than a preset contribution threshold are subjected to initial sparsification. According to the preset training step size period, the gradient change magnitude of each weight in the specified projection layer is recalculated, and the current sparsified weight range is updated according to the gradient change magnitude. Among them, the weights whose gradient contribution is still lower than the preset contribution threshold are retained in the sparsified state, and newly added weights whose gradient contribution is lower than the preset contribution threshold are added to the sparsified range. For weights that are already in a sparse state, the gradient change of the weights is continuously monitored. When the gradient contribution of the weights is detected to rise to the preset activation standard, the sparse state of the weights is removed and they are reactivated so that the weights can participate in the model training process.

[0010] In one possible implementation, when a new scenario is identified, a target scenario expert model that is relevant to the new scenario in terms of scenario attributes or business characteristics is selected from the candidate scenario expert models, and transfer learning is performed based on the target scenario expert model to complete the new scenario adaptation, including: Identify the scenario attributes and business characteristics of the new scenario, and based on the scenario attributes and business characteristics, select target scenario expert models whose relevance meets a preset relevance threshold from each candidate scenario expert model; Load the weight parameters of the expert model for the target scene, freeze the specified projection layer parameters responsible for the attention mechanism in the weight parameters, and retain the general features already learned by the model; The training learning rate is adjusted, and based on the scene training data corresponding to the new scene, the target scene expert model is fine-tuned through model feature sharing and knowledge transfer to form a scene expert model corresponding to the new scene, thus completing the adaptation to the new scene.

[0011] In one possible implementation, adjusting the training learning rate involves fine-tuning the target scene expert model based on the scene training data corresponding to the new scene, through model feature sharing and knowledge transfer, to form a scene expert model corresponding to the new scene, thus completing the new scene adaptation. This includes: The target learning rate for fine-tuning the new scene is determined based on the original learning rate of the expert model for the target scene, and the target learning rate is lower than the original learning rate. The scene training data corresponding to the new scene is input into the target scene expert model in a preset batch. During the fine-tuning process, the parameters of the frozen layer remain unchanged, and only the parameters of the trainable layer are updated. Through iterative training, the loss value of the target scene expert model on the new scene training data converges to a preset range. With the help of model feature sharing and knowledge transfer, the model learns the special features of the new scene and completes the adaptation to the new scene.

[0012] Furthermore, to achieve the above objectives, this application also proposes an operational performance evaluation device, which includes: The acquisition module is used to acquire the target scenario and evaluation time range configured by the user. The evaluation module is used to input the target scenario and the evaluation time interval into the operation effect evaluation model to obtain the operation effect evaluation summary output by the operation effect evaluation model. The operation effect evaluation model consists of a single base model and multiple scenario-specific adaptation modules. The operation effect evaluation model is obtained by performing multi-scenario-specific fine-tuning training on scenario training data.

[0013] In addition, to achieve the above objectives, this application also proposes an operational performance evaluation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the operational performance evaluation method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the operational performance evaluation method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the operational performance evaluation method described above.

[0016] This application provides an operational effectiveness evaluation method, apparatus, device, and storage medium. The operational effectiveness evaluation method obtains the target scenario and evaluation time interval configured by the user, and then inputs the target scenario and evaluation time interval into the operational effectiveness evaluation model to obtain the operational effectiveness evaluation summary output by the operational effectiveness evaluation model. The operational effectiveness evaluation model consists of a single base model and multiple scenario-specific adaptation modules. The operational effectiveness evaluation model is obtained by performing multi-scenario-specific fine-tuning training on scenario training data. By setting a single base and multiple adaptation module architecture, data interference between scenarios is avoided, and the model accurately adapts to the specific needs of each scenario. Furthermore, it eliminates the need to deploy multiple independent models, significantly saving computational and storage resources for training and inference. The multi-scenario-specific fine-tuning ensures the scenario adaptability and accuracy of the evaluation summary. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the operational effectiveness evaluation method of this application. Figure 2 A simplified flowchart illustrating the operational effectiveness evaluation method used in this application; Figure 3 This is a schematic diagram of the module structure of the operation effect evaluation device according to an embodiment of this application; Figure 4 This is a schematic diagram of the equipment structure of the hardware operating environment involved in the operational effectiveness evaluation method in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, big data service platform, or operational performance evaluation system capable of performing the above functions. The following description uses an operational performance evaluation system as an example to illustrate this embodiment and the subsequent embodiments.

[0024] Based on this, the embodiments of this application provide a method for evaluating operational effectiveness, referring to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the operational effectiveness evaluation method of this application.

[0025] In this embodiment, the operational effectiveness evaluation method includes steps S11-S12: Step S11: Obtain the target scenario and evaluation time range configured by the user; It's important to note that the target scenario refers to the specific business scenario where the user needs to evaluate operational effectiveness, covering various sub-scenarios within the life services sector, such as hotel booking, visa services, new home viewing, and utility payment. The evaluation timeframe refers to the specific time range set by the user for conducting the operational effectiveness evaluation. This range can be flexibly configured according to business needs, such as a single day, week, month, or any custom time period. The core purpose of this step is to clarify the object and timeframe of the operational effectiveness evaluation, ensuring that the evaluation results accurately correspond to the scenarios and time dimensions that users are concerned with, thus meeting the business needs of evaluating the operational effectiveness of scenario-based products, optimizing resource allocation, and improving user experience.

[0026] In one possible implementation, users can select a target scenario through the system's visual configuration interface, set the start and end times of the evaluation time interval using a date selector, and the system automatically verifies the rationality of the time interval to avoid invalid configurations where the start time is later than the end time.

[0027] In addition, in this embodiment of the application, the methods for obtaining user configuration information include, but are not limited to, user active input, drop-down selection, batch import, etc. The system will perform legality verification on the target scenario configured by the user to confirm that the scenario belongs to the business scenario range supported by the system. At the same time, the system will perform format verification and validity judgment on the evaluation time interval to ensure that the time information is complete and the format is standardized.

[0028] Specifically, the system provides multi-dimensional configuration entry points for users to operate. Users can choose the corresponding method to configure the target scenario and evaluation time interval according to their actual needs. After receiving the user's configuration information, the system first verifies the legality of the target scenario by querying the scenario configuration library to confirm whether the scenario is a valid business scenario. Then, it verifies the evaluation time interval, checking whether the time format conforms to the preset standard and whether the start time is earlier than the end time. After the verification is passed, the system records and stores the user's configured target scenario and evaluation time interval to form standardized input data, which is ready for subsequent model calls.

[0029] For example, if a user needs to evaluate the operational effectiveness of the "hotel booking scenario" in December 2024, they can select "hotel booking scenario" as the target scenario through the scenario drop-down menu in the system configuration interface, and set the start time of the evaluation time interval to December 1, 2024 and the end time to December 31, 2024 through the date selector. After receiving the configuration information, the system verifies and confirms that the "hotel booking scenario" is a valid scenario supported by the system, and that the time interval format is standardized and logically reasonable, and then stores the configuration information.

[0030] Step S12: Input the target scenario and the evaluation time interval into the operation effect evaluation model to obtain the operation effect evaluation summary output by the operation effect evaluation model. The operation effect evaluation model consists of a single base model and multiple scenario-specific adaptation modules. The operation effect evaluation model is obtained by performing multi-scenario-specific fine-tuning training on scenario training data.

[0031] It should be noted that the operational performance evaluation model refers to an intelligent model capable of analyzing and summarizing operational performance across multiple scenarios. Based on the input target scenario and evaluation time interval, it can output a corresponding operational performance evaluation summary. The single-base model refers to the pre-trained model that serves as the basic framework of the model. In this embodiment, the Qwen2.5-3B-Instruct model is selected, which provides the model with general language understanding and generation capabilities. The scenario-specific adaptation module refers to a dedicated adaptation component trained for a specific business scenario, namely the LoRA Adapter, which enables the model to accurately adapt to the data distribution, business objectives, and text style of the corresponding scenario. Scenario training data refers to high-quality training data adapted to various business scenarios, including scenario background information, historical user behavior data, and daily user behavior statistics.

[0032] Furthermore, multi-scenario specialized fine-tuning training refers to the targeted fine-tuning process of a single base model based on training data for different business scenarios, aiming to enable the model to master the unique characteristics of each scenario. Operational performance evaluation summary refers to the structured summary report output by the model, containing key information on the operational performance of the target scenario within the evaluation time period, covering core evaluation content such as exposure, click-through rate, recommendation strategy effectiveness, and user feedback.

[0033] The purpose of this step is to leverage an operational performance evaluation model trained with multi-scenario fine-tuning to quickly and accurately analyze the operational performance of a target scenario within a specified time interval, generating a comprehensive and targeted evaluation summary. This also addresses the issues of interference between scenarios and the difficulty in considering the specific characteristics of each scenario in traditional unified models. In one possible implementation, after receiving input data, the model automatically matches the scenario-specific adaptation module corresponding to the target scenario, loads the module's weight parameters, and combines the general capabilities of a single base model to analyze and summarize the scenario's operational data within the evaluation time interval, generating an evaluation report adapted to the business characteristics of that scenario.

[0034] In addition, in this embodiment, the training process of the operational performance evaluation model integrates model quantization, gradient accumulation, and dynamic sparsity mechanism of model parameters. Model quantization refers to quantizing the base model parameters from float16 precision to 4-bit to reduce memory usage. Gradient accumulation refers to breaking down a batch of training into multiple smaller batches for training, accumulating gradients, and then updating the model parameters to further reduce memory consumption. The dynamic sparsity mechanism of model parameters refers to sparsifying the weights with low contribution during the training process and reactivating the sparsified weights that meet the conditions to achieve continuous optimization of memory usage.

[0035] Specifically, the system invokes the operational effectiveness evaluation model, inputting the standardized target scenario and evaluation time interval into the model. The model first matches the corresponding module from multiple scenario-specific adaptation modules based on the target scenario's identifier information, and loads the module's training weights. Then, combining the general language processing capabilities of the single-base model, it calls relevant data from the scenario background knowledge base and behavior vector library to perform multi-dimensional analysis of the operational data of the target scenario within the evaluation time interval, including data trend analysis, strategy effectiveness evaluation, and user behavior feature extraction. After the analysis is completed, the model generates an operational effectiveness evaluation summary containing core evaluation indicators, key conclusions, and optimization suggestions according to a preset structured format, and outputs it to the user.

[0036] For example, the "Visa Service Scenario" and the evaluation timeframe "November 1, 2024 - November 30, 2024" are input into the operational performance evaluation model. The model matches and loads the scenario-specific adaptation module corresponding to the "Visa Service Scenario," and, combined with the capabilities of the Qwen2.5-3B-Instruct base model, retrieves information such as the business description and recommendation strategies of the "Visa Service Scenario" from the scenario background knowledge base, as well as relevant data such as the exposure, click-through rate, and application success rate of the scenario within the evaluation timeframe from the behavior vector library. After multi-dimensional analysis, an operational performance evaluation summary is generated, including core content such as "The average exposure of the visa service scenario in November 2024 was 315,141.6, the click-through rate increased by 5% compared to the previous month, the application success rate of the core recommendation strategy A was the highest, and it is recommended to continuously optimize the reach of strategy A," and this summary is then output to the user.

[0037] This embodiment first obtains the target scenario and evaluation time interval configured by the user, clarifies the evaluation object and scope, and then inputs them into an operational effect evaluation model composed of a single base model and multiple scenario-specific adaptation modules to obtain a targeted operational effect evaluation summary. By leveraging the model trained with multi-scenario-specific fine-tuning, it effectively solves the problems of interference between scenarios and insufficient specific adaptation in traditional unified models. At the same time, the model optimization mechanism reduces resource consumption and can quickly and accurately output evaluation summaries that meet business needs, providing a reliable decision-making basis for optimizing the operational effect of scenario products, rationally allocating resources, and improving user experience.

[0038] In one feasible implementation, before inputting the target scenario and the evaluation time interval into the operational performance evaluation model to obtain the operational performance evaluation summary output by the operational performance evaluation model, the method further includes: Step S21: Construct scene training data adapted to multiple scenarios; It's important to note that scenario training data refers to high-quality data adapted to various business scenarios and used for model fine-tuning. This data supports the model in learning the specific features of each scenario, ensuring accurate output results in the operational performance evaluation of the corresponding scenario. Multiple scenarios refer to various sub-scenarios covering the life services field, such as hotel booking, visa services, new home viewing, and utility payment. The core purpose of this step is to address the problems of low-quality training data and poor scenario adaptability in traditional models. By integrating scenario background information and user behavior data, and leveraging retrieval-enhanced generation techniques, training data tailored to the business characteristics of each scenario is generated, ensuring that the model can accurately understand the business objectives, data distribution, and text style of different scenarios.

[0039] In one possible implementation, the generation of scene training data can be automatically updated at fixed intervals, combining the latest scene background changes and user behavior data to continuously optimize the quality of training data and improve the adaptability of the model.

[0040] Specifically, the system first acquires the scenario background information and historical user behavior data corresponding to each business scenario, and then structures and organizes the scenario background information to build a scenario background knowledge base. The historical user behavior data is converted into vector form using embedding technology and stored in a vector database to form a behavior vector library. Next, the system acquires the daily user behavior statistics corresponding to each business scenario, generates retrieval vectors based on this data, and retrieves historical behavior data related to the current scenario and current time characteristics from the behavior vector library. Finally, through retrieval enhancement generation technology, the retrieved relevant historical behavior data and the corresponding scenario background information from the scenario background knowledge base are input into a general large model. The general large model then generates scenario training data adapted to the business scenario, ensuring that the training data includes both scenario background cognition and user behavior patterns.

[0041] Step S22: Based on the scene training data, perform multi-scene-specific fine-tuning training on the single base model, and optimize the training process by integrating model quantization, gradient accumulation and dynamic sparsity mechanism of model parameters to form an operation effect evaluation model equipped with multiple scene-specific adaptation modules. The operation effect evaluation model includes the candidate scene expert model that has been trained for each scene. It should be noted that the dynamic sparsity mechanism of model parameters refers to the sparsification of weights with low contribution in the model during training (weights reset to 0, gradient frozen), and the dynamic adjustment mechanism of reactivating the sparsified weights whose gradient contribution is subsequently increased. Candidate scenario expert models refer to model instances trained for each scenario in the operational effectiveness evaluation model, capable of independently handling the evaluation task of that scenario. The purpose of this step is to solve the problems of high training resource consumption and mutual interference between scenarios in traditional multi-scenario models. It achieves adaptation of a single base model to multiple scenarios through S-LoRA (Serving thousands of Concurrent Low-Rank Adaptation) technology, combined with various optimization mechanisms to reduce memory usage and computational costs during training, while ensuring the specificity and accuracy of each scenario expert model.

[0042] In one possible implementation, multi-scenario-specific fine-tuning training can be conducted in parallel, simultaneously training dedicated adaptation modules for multiple scenarios, thus significantly shortening the overall training cycle.

[0043] Specifically, the system first performs parameter quantization on the single base model, quantizing its parameters from float16 precision to 4-bit to reduce the model's basic GPU memory usage. For each business scenario, it obtains the corresponding scenario training data and performs quantization on the single base model's q_proj, k_proj, v_proj, and o_proj. The scene-specific adaptation module corresponding to the specified projection layer is trained. During training, a gradient accumulation method is used to break down a batch of training into multiple smaller batches, and the model parameters are updated after the gradients are accumulated. At the same time, a dynamic sparsity mechanism for model parameters is set. In the initial stage of training, the weight gradient of the specified projection layer is globally sampled, and the weights with gradient contribution below the preset threshold are initially sparsified. Then, the sparsity range is updated according to the preset training step size period. The weights with low contribution are retained in the sparsity state, and new low contribution weights are added to the sparsity range. The sparsified weights with significantly improved contribution are reactivated. After all scene-specific adaptation modules are trained, each module is integrated with a single base model to form an operational effect evaluation model that includes expert models of candidate scenes for each scene, so as to achieve coverage of multi-scene evaluation needs by a single model.

[0044] Step S23: When a new scenario is identified, select a target scenario expert model that is related to the new scenario in terms of scenario attributes or business characteristics from the candidate scenario expert models, and perform transfer learning based on the target scenario expert model to complete the adaptation of the new scenario.

[0045] It's important to note that "new scenario" refers to entirely new business scenarios that haven't undergone prior model training and adaptation, such as newly added "pet service scenarios" or "housekeeping service scenarios." These scenarios typically face challenges like scarce training data and difficulties in cold start. Scenario attributes refer to attributes describing the essential characteristics of the scenario, such as industry category, service type, and target user group. Business characteristics refer to the core business features exhibited by the scenario during operation, such as recommendation strategies, data metric types, and user interaction methods. The target scenario expert model refers to a trained model selected from candidate scenario expert models that is highly correlated with the scenario attributes or business characteristics of the new scenario; it serves as the foundational model for transfer learning.

[0046] Furthermore, transfer learning refers to a learning method that transfers knowledge learned by a trained model to new tasks or scenarios, reducing training costs and data requirements for new scenarios by reusing existing knowledge. The core purpose of this step is to address the problems of poor model training performance caused by cold starts in new scenarios and data sparsity in low-traffic scenarios. By reusing the knowledge and weight parameters of relevant scenario expert models, only a few parameters need to be fine-tuned to quickly adapt to new scenarios, significantly reducing training costs and shortening the training cycle, while ensuring the quality and stability of operational performance evaluation in new scenarios.

[0047] In one possible implementation, the screening of scenario relevance can adopt a multi-dimensional scoring mechanism, which calculates the relevance scores of candidate models to new scenarios from multiple dimensions such as industry similarity, user group similarity, and business process similarity, and selects the model with the highest score as the expert model for the target scenario.

[0048] Specifically, the system first identifies the scenario attributes and business characteristics of the new scenario. For example, if the new scenario is a "pet service scenario," its scenario attribute is a life service category, its target users are pet owners, and its business characteristics include pet supply recommendations and pet service bookings. Based on these characteristics, the system calculates the correlation between each candidate scenario expert model and the new scenario, and selects target scenario expert models with a correlation higher than a preset correlation threshold. For example, the "convenience service scenario" expert model is selected as the target model. Then, the system loads the weight parameters of the target scenario expert model, freezes the parameters of the q_proj and k_proj layers responsible for the attention mechanism, and retains the general features already learned by the model. The training learning rate is adjusted to 1 / 4 of the original expert model's learning rate, and the model is fine-tuned based on a small amount of training data corresponding to the new scenario, updating only the parameters of the v_proj and o_proj layers. Through iterative training, the model's loss value on the new scenario training data converges to a preset range. With the help of model feature sharing and knowledge transfer, the model quickly masters the specific features of the new scenario, forming a scenario expert model corresponding to the new scenario, thus completing the new scenario adaptation. (See reference...) Figure 2 .

[0049] This embodiment first constructs high-quality scenario training data adapted to multiple scenarios, then performs multi-scenario-specific fine-tuning on a single base model based on this data, integrates multiple optimization mechanisms to form an operational performance evaluation model, and finally rapidly adapts to new scenarios through transfer learning. This effectively solves the problems of severe interference between scenarios, high resource consumption, and difficulty in cold-starting new scenarios in traditional multi-scenario models. By using S-LoRA technology to achieve single-model adaptation to multiple scenarios and reusing existing knowledge through transfer learning, it not only ensures the accuracy and specificity of evaluation for each scenario but also significantly reduces training and deployment costs, providing efficient and reliable technical support for multi-scenario business operation performance evaluation.

[0050] In one feasible implementation, constructing scene training data adapted to multiple scenarios includes: Step S31: Obtain the scenario background information and historical user behavior data corresponding to each business scenario; It's important to clarify that "business scenarios" refer to the various specific service scenarios involved in the operation process, covering multiple sub-scenarios within the life services field, such as hotel booking, visa services, new home viewing, and utility payment. Scenario background information refers to the set of information that describes the core characteristics of each business scenario, including scenario product functions, recommendation strategies, target user groups, business processes, and service scope; it forms the basis for the model's understanding of scenario characteristics. Historical user behavior data refers to user interaction-related data accumulated by each business scenario over past operational cycles, specifically including statistical data such as exposure, clicks, conversion rates, user operation records, and business completion success rates. This data reflects user behavior patterns and the operational status of the scenario.

[0051] In one possible implementation, data acquisition can be achieved by combining timed batch acquisition with real-time incremental acquisition. Timed batch acquisition is used to acquire historical accumulated data, while real-time incremental acquisition is used to supplement the latest data, ensuring the integrity and timeliness of the data.

[0052] Specifically, the system extracts scenario background information corresponding to each business scenario from the business system database through a preset data collection interface. This includes structured and unstructured information such as official descriptions of the scenarios, service details, and recommendation strategy documents. At the same time, it extracts historical user behavior data for each scenario from the user behavior log database, covering data such as exposure statistics, click records, user operation trajectories, and business conversion results within a specified time period. The system performs preliminary verification on the acquired scenario background information and historical user behavior data, removing invalid, duplicate, and incorrectly formatted data to ensure the authenticity and usability of the data, thus forming a standardized raw dataset.

[0053] For example, for the "visa service scenario," the system obtains background information about the scenario from the business system, including "visa application services covering major countries and regions around the world, providing guidance on material preparation, assistance in filling out application forms, professional customer service support, and simplifying the visa application process." It also extracts historical user behavior data for the past 6 months from user behavior logs, including average daily exposure, clicks, visa inquiries for different countries, and visa application success rates. This data is then validated, and abnormal exposure data caused by system failures and duplicate user inquiry data are removed to form the original dataset for the scenario.

[0054] Step S32: Construct a scene background knowledge base based on the scene background information, convert the historical user behavior data into vector form and store it, and construct a behavior vector library; It's important to note that the scenario background knowledge base refers to a structured database specifically designed to store background information for various business scenarios. It provides models with fundamental scenario-related knowledge, enabling them to quickly retrieve scenario feature information during training and inference. The behavior vector library refers to a database that stores historical user behavior data in vector form. This vector form transforms high-dimensional, discrete user behavior data into low-dimensional, continuous, computable vectors, facilitating rapid retrieval through similarity calculations. The vector form refers to the numerical vectors formed by mapping the original data to a high-dimensional vector space using embedding technology, preserving the core features and relationships of the original data.

[0055] The purpose of this step is to solve the problems of messy original data formats and low retrieval efficiency. By building a structured scenario background knowledge base and behavior vector library, we can achieve standardized data storage and efficient retrieval, while reducing resource consumption during data storage and retrieval.

[0056] In one possible implementation, the scene background knowledge base is stored using a relational database, with data tables established according to scene categories. Each piece of background information corresponds to a unique scene identifier and keyword index, facilitating quick retrieval. The behavior vector library is stored using a dedicated vector database, supporting efficient vector similarity calculation and retrieval operations.

[0057] Specifically, the system performs structured processing on the pre-verified scene background information, splitting and organizing it according to preset fields such as scene identifier, scene name, product function, recommendation strategy, target user, and service scope. It establishes the association between scene background information and scene identifier, stores the structured information in a relational database, constructs a scene background knowledge base, and builds a keyword index to improve retrieval speed. Simultaneously, it calls a preset embedding model to convert standardized historical user behavior data into fixed-dimensional vectors, preserving user behavior characteristics and correlation patterns in the data. The converted vector data is then associated with corresponding scene identifiers, timestamps, and other metadata and stored in a vector database to construct a behavior vector library, supporting fast retrieval operations based on vector similarity.

[0058] For example, the system organizes the structured background information of the "visa service scenario" according to preset fields and stores it in the scenario background knowledge base, and establishes keyword indexes such as "visa application assistance", "material guidance" and "customer service support"; inputs historical user behavior data such as the average daily exposure, click volume and visa application success rate of the scenario over the past 6 months into the embedding model, converts it into 128-dimensional vector data, with each vector corresponding to the core feature of a historical data point; associates these vector data with metadata such as the scenario identifier "visa service" and the time period to which the data belongs, and stores them in the vector database to complete the behavior vector library of the scenario.

[0059] Step S33: Obtain daily user behavior statistics for each of the aforementioned business scenarios, and retrieve relevant historical behavior data from the behavior vector library based on the daily user behavior statistics. It should be noted that the daily user behavior statistics refer to the user behavior statistics generated by each business scenario on the current evaluation day, including real-time statistics such as average daily exposure, clicks, conversion rate, and popular service inquiries. These data reflect the operational status and user behavior trends of the scenario on that day. Relevant historical behavior data refers to historical user behavior data in the behavior vector library that are similar in features to the daily user behavior statistics. By retrieving this data, historical references can be provided for generating training data for the current scenario.

[0060] The purpose of this step is to use the features of real-time user behavior data of the day to quickly filter out historical data with similar features from the behavior vector library. This ensures that the subsequently generated scenario training data includes both the real-time operational status of the day and the operational experience of similar historical scenarios, thus ensuring that the training data can balance timeliness and reference value and improve the model's ability to adapt to dynamic changes in scenarios.

[0061] In one possible implementation, the retrieval process can adopt a hierarchical retrieval strategy. First, historical vector data corresponding to the scene is filtered out by scene identifier, and then data with similar features to the data of the day is filtered out based on vector similarity calculation, thereby improving retrieval efficiency and accuracy.

[0062] Specifically, the system acquires daily user behavior statistics for each business scenario through a real-time data acquisition interface, including average exposure, clicks, conversion rate, user inquiry hotspots, and business processing type distribution data up to the present. It then standardizes the daily user behavior statistics, converting them into retrieval vectors with the same format as those in the behavior vector library. Based on scenario identifiers, it filters all historical vector data corresponding to the current business scenario from the behavior vector library. Finally, it uses a cosine similarity algorithm to calculate the similarity between the retrieval vector and each historical vector data, sets a similarity threshold, and filters out the original historical user behavior data corresponding to historical vector data with similarity higher than the threshold, using this as historical behavior data related to the user behavior of the day.

[0063] For example, the user behavior statistics for the "Visa Service Scenario" on that day were "average exposure of 315,141.6, click-through rate of 8%, and popular countries of inquiry were the United States, Japan, and Thailand." The system standardized this data and converted it into a retrieval vector. It then selected all historical vector data corresponding to the "Visa Service Scenario" from the behavior vector library. The system calculated the cosine similarity between the retrieval vector and each historical vector, set a similarity threshold of 0.8, and selected historical vector data with similarity scores higher than 0.8. The corresponding original historical user behavior data were historical data with similar characteristics, such as "average exposure of 308,920 on a certain historical day, click-through rate of 7.8%, and popular countries of inquiry were the United States, Japan, and South Korea."

[0064] Step S34: Using retrieval enhancement generation technology, the retrieved relevant historical behavior data and the corresponding scene background information in the scene background knowledge base are input into the general large model to generate scene training data adapted to each of the business scenarios.

[0065] It's important to note that Retrieval Augmentation (RAG) technology combines data retrieval with language generation capabilities. Its core logic involves first retrieving external data relevant to the current task, then using this data as contextual input to the generation model, assisting the model in generating more accurate and relevant content. General-purpose large models, on the other hand, refer to pre-trained models with powerful general language understanding and generation capabilities, capable of generating logically coherent and detailed text data based on the input context.

[0066] The purpose of this step is to address the lack of scenario-specificity and data support in traditional model training data generation. By using retrieval-enhanced generation technology, it combines scenario background information with historical data related to user behavior on the current day to guide the generation of scenario training data for a general-purpose model that conforms to the scenario's business logic and reflects user behavior patterns. In one possible implementation, a structured prompt template can be set when inputting the general-purpose model to clarify the generation format, core elements, and content requirements of the training data, ensuring that the generated scenario training data has a uniform format and highlights key points.

[0067] Specifically, the system first retrieves the scenario background information corresponding to the current business scenario from the scenario background knowledge base, including core content such as scenario product functions, recommendation strategies, and service scope; it then integrates the retrieved relevant historical behavior data with the scenario background information in a preset format to form a structured input context; it calls a general large model, inputting the integrated input context along with a preset prompt word template into the model. The prompt word template explicitly requires the model to generate scenario training data based on the scenario background information and historical behavior data, including scenario operation status analysis, user behavior feature summary, and recommendation strategy optimization suggestions; finally, it receives the text data output by the general large model, performs format verification and content review to ensure that the data meets the training requirements, and forms the final scenario training data.

[0068] For example, for the "visa service scenario," the system retrieves background information from the scenario background knowledge base and combines it with relevant historical behavioral data to form an input context: "Scenario Background: Visa application services covering major countries and regions worldwide, providing guidance on material preparation, assistance in filling out application forms, and professional customer service support; Relevant Historical Behavioral Data: Average daily exposure of 308,920, click-through rate of 7.8%, popular countries of inquiry are the United States, Japan, and South Korea; Daily User Behavior Statistics: Average daily exposure of 315,141.6, click-through rate of 8%, popular countries of inquiry are the United States, Japan, and Thailand." This context, along with the prompt template "Based on scenario background, historical behavioral data, and daily user behavior statistics, generate a summary of the scenario's operational performance, including core data, user behavior characteristics, and recommendation strategy suggestions," is input into the general model. The model outputs: "The average daily exposure for the visa service scenario is 315,141.6, with a click-through rate of 8%, an improvement of 0.2 compared to similar historical dates." The percentage point; popular user inquiries are concentrated on visa processing for the United States, Japan, and Thailand. It is recommended that the recommendation strategy focus on visa processing services for these three countries, while optimizing the push logic of the material preparation guidance document to improve user conversion efficiency.

[0069] This embodiment first acquires the background information and historical user behavior data of each business scenario, then constructs a structured scenario background knowledge base and behavior vector library, subsequently retrieves relevant historical behavior data based on the daily user behavior statistics, and finally generates training data adapted to each scenario through retrieval enhancement generation technology. This effectively solves the problems of low quality, poor scenario adaptability, and lack of data support in traditional scenario training data. By integrating scenario background and user behavior data, and leveraging retrieval enhancement generation technology, the professionalism, relevance, and timeliness of the training data are ensured. At the same time, the structured knowledge base and vector library design also improves the efficiency of data storage and retrieval, and reduces resource consumption.

[0070] In one feasible implementation, based on the scene training data, the single base model undergoes multi-scene-specific fine-tuning training, integrating model quantization, gradient accumulation, and dynamic sparsity mechanisms for model parameters to optimize the training process, forming an operational performance evaluation model equipped with multiple scene-specific adaptation modules, including: Step S41: Perform parameter quantization processing on the single base model; It should be noted that, to conserve computational resources for general-purpose large models, this embodiment uses a small-sized expert model instead of a large-sized general-purpose model. Although LoRA fine-tuning only fine-tunes some parameters compared to full-parameter fine-tuning, significantly reducing the resources and time required for fine-tuning, it can lead to interference between data from different domains when dealing with multi-scenario problems. To adapt to multi-scenario problems in real life, this embodiment adopts the S-LoRA fine-tuning deployment scheme. S-LoRA (Serving thousands of Concurrent Low-Rank Adaptation) is a technology optimized based on LoRA. After fine-tuning multiple domains of the same base model, S-LoRA only requires deploying one application. The base model is loaded into the GPU, and then, according to different task requests, the corresponding LoRA adapter is loaded from main memory into the GPU, thereby achieving lightweight multi-expert model switching and parallelism, enabling the model to handle multiple requests for different specific tasks simultaneously.

[0071] Furthermore, parameter quantization refers to a technique that reduces the data precision of model parameters to decrease the model's storage space and computational resource consumption while ensuring that model performance is not significantly affected. The core purpose of this step is to solve the problem of large number of parameters and high memory consumption in a single base model. By compressing the model size through quantization, memory resources are reserved for the parallel training and deployment of subsequent multi-scenario-specific adaptation modules, while improving the model's inference speed and reducing the overall computational cost.

[0072] In one possible implementation, the quantization process can select different quantization precisions according to the business requirements for model performance, such as 2-bit, 4-bit, 8-bit, etc. The lower the precision, the less memory is used, but the loss of model performance needs to be balanced.

[0073] Specifically, the system loads the preset single-pedestal model Qwen2.5-3B-Instruct, calls a dedicated model quantization tool to convert the model's original float16 precision parameters to 4-bit precision. During the quantization process, a linear quantization algorithm is used to map and compress the parameters. At the same time, the quantized model is calibrated using a calibration dataset to ensure that the loss of core capabilities such as language understanding and text generation is controlled within a preset range. After quantization, the model is verified for integrity to confirm that all parameters have been converted to the correct precision, with no parameter loss or format errors, thus forming the quantized single-pedestal model.

[0074] For example, the system loads the Qwen2.5-3B-Instruct base model, which originally uses about 6GB of GPU memory at float16 precision. By converting its parameters to 4-bit precision using a quantization tool, the performance of the quantized model is tested using a calibration dataset. The results show that its accuracy in the scene summary task only decreases by 0.5%, which meets the business requirements. The GPU memory usage of the quantized model is reduced to about 1.5GB, which significantly reduces GPU memory consumption and provides sufficient resources for the training of subsequent scene-specific adaptation modules.

[0075] Step S42: For any business scenario, based on the scenario training data corresponding to the business scenario, train the scenario-specific adaptation module corresponding to the business scenario on the specified projection layer of the single base model. During the training process, the training batch is split by gradient accumulation and a dynamic sparsity mechanism for model parameters is set to sparsify the weights with low contribution in the model and reactivate the sparsified weights that meet the activation conditions. It should be noted that the designated projection layer refers to the network layer used for vector projection in a single pedestal model. In this embodiment, it specifically refers to the q_proj layer, k_proj layer, v_proj layer, and o_proj layer. These layers are the core target layers for LoRA fine-tuning, capable of accurately adapting to scene-specific features. Gradient accumulation refers to breaking down a complete training batch into multiple smaller batches for training, accumulating the gradients of multiple smaller batches, and then updating the model parameters, thereby reducing the GPU memory consumption of a single training session. A training batch refers to the set of data input during a single training session. The dynamic sparsity mechanism of model parameters refers to the dynamic adjustment of the sparsity state of the model weights during training. Weights with low contribution are sparsified (weights reset to 0, gradients frozen), and the previously sparsified weights with increased contribution are reactivated. Contribution refers to the degree of influence of the model weights on the scene training task and scene adaptation effect, which is quantified by the magnitude of the gradient change of the weights. Sparsification processing refers to setting the weights with low contribution to 0 and freezing their gradients, so that they no longer participate in the model training update operation. The activation condition refers to the gradient contribution of the sparsed weights increasing to a preset standard, making them valuable for retraining the model.

[0076] The purpose of this step is to address the issues of inter-scene interference and excessive memory consumption during multi-scene training. By training scene-specific adaptation modules at designated projection layers, the accurate acquisition of scene-specific features is ensured. Simultaneously, gradient accumulation and dynamic sparsity mechanisms for model parameters further reduce memory consumption during training, enabling efficient parallel training of multi-scene adaptation modules. In one possible implementation, scene-specific adaptation modules for different business scenarios can be trained in parallel, simultaneously on a quantized single base model, significantly shortening the overall training cycle.

[0077] Specifically, in one embodiment, for any given business scenario, the system acquires the corresponding scenario training data, preprocesses it according to a preset format, and inputs it into a quantized single-base model. The system initializes the weight parameters of the scenario-specific adaptation module on the q_proj, k_proj, v_proj, and o_proj layers of the model, and sets the LoRA training hyperparameters (r=8, alpha=16, learning_rate=1e-4, batch_size=8). During training, a gradient accumulation method is used to decompose a batch of training into four smaller batches, and the gradients of these four smaller batches are accumulated before updating the model parameters. Simultaneously, a dynamic sparsity mechanism for model parameters is activated. In the initial training phase, the weight gradients of the specified projection layer are globally sampled, and the 20% with the smallest absolute gradient values ​​are selected as the initial sparsity positions and subjected to sparsification. Every 100 training steps constitute an update cycle, and the gradient change amplitude of the weights is recalculated, with the current sparsity rate set to 10%. The sparsity range of the weights is updated, and new low-contribution weights are added to enter the sparsity state. The already sparsified weights are continuously monitored. If their gradient contribution increases significantly and they enter the top 10%, the sparsity state is deactivated and they are reactivated to participate in subsequent training.

[0078] For example, for the "visa service scenario," the system acquires the scenario training data, preprocesses it, and inputs it into the quantized Qwen2.5-3B-Instruct model. It initializes scenario-specific adaptation modules in the q_proj, k_proj, v_proj, and o_proj layers of the model, setting the corresponding LoRA hyperparameters. The training batch with batch_size=8 is split into 4 smaller batches, and gradients are accumulated after each smaller batch is trained. Parameters are updated after all 4 smaller batches are completed. In the initial training phase, the 20% of weights with the smallest absolute gradient values ​​in the specified projection layer are sparsified. At the 100th training step, the weight gradients are recalculated, and the newly added 10% of low-contribution weights are added to the sparsification range. Simultaneously, 3 weights with gradient contributions that are already in the top 10% of the sparsified weights are activated. Through this mechanism, the memory usage during training is further reduced by 30%, and the model can accurately learn the business characteristics of the visa service scenario.

[0079] Step S43: After completing the training of all scenario-specific adaptation modules, integrate each scenario-specific adaptation module with a single base model to form an operational performance evaluation model that includes candidate scenario expert models corresponding to each business scenario.

[0080] It should be noted that integration refers to associating and binding all trained scene-specific adaptation modules with a single base model, enabling the model to automatically load the corresponding scene-specific adaptation module based on different scene requests. A candidate scene expert model refers to a model instance formed by combining the single base model with a scene-specific adaptation module after integration, capable of independently handling the operational effectiveness evaluation task for that scene. An operational effectiveness evaluation model refers to a composite model integrating the single base model and all scene-specific adaptation modules, possessing the ability to handle operational effectiveness evaluation needs across multiple scenarios and automatically switching to the corresponding candidate scene expert model based on the target scene.

[0081] The purpose of this step is to achieve full coverage and adaptation of a single model to multiple scenarios, solving the problem that traditional multi-scenario models need to be deployed separately and consume a lot of resources. By integrating the operational performance evaluation model, only one base model and multiple lightweight adaptation modules need to be deployed to support the operational performance evaluation of all business scenarios, which can significantly save deployment resources and maintenance costs, while ensuring the accuracy of evaluation in each scenario.

[0082] In one possible implementation, a mapping table between scene identifiers and scene-specific adaptation modules is established during the integration process. When a scene evaluation request is received, the model can quickly locate and load the corresponding adaptation module based on the scene identifier, thereby improving the response speed.

[0083] Specifically, after the training of scenario-specific adaptation modules for all business scenarios is completed, the system collects the training weight parameters and performance evaluation reports of each adaptation module to verify that the evaluation accuracy, loss value, and other indicators of each adaptation module in the corresponding scenario meet the preset standards; establishes a one-to-one mapping relationship between scenario identifiers and scenario-specific adaptation modules to form a mapping relationship table; stores the weight parameters and mapping relationship table of all scenario-specific adaptation modules in a designated directory and associates them with the quantized single base model; when receiving an operational effect evaluation request for a certain business scenario, the system queries the mapping relationship table according to the scenario identifier, quickly loads the corresponding scenario-specific adaptation module, and combines it with the single base model to form a candidate scenario expert model corresponding to that scenario to complete the evaluation task; the candidate scenario expert models of all scenarios are combined to form the final operational effect evaluation model, realizing unified deployment and flexible invocation of multiple scenarios.

[0084] For example, the system completes the training of dedicated adaptation modules for four scenarios: hotel booking, visa services, utility bill payment, and new home viewing. The evaluation accuracy of each module in the corresponding scenario reaches over 95%. It establishes mapping relationships such as "hotel" corresponding to the hotel booking adaptation module and "visa" corresponding to the visa service adaptation module. The weight parameters and mapping relationship table of the four adaptation modules are stored in the model directory and integrated with the quantized Qwen2.5-3B-Instruct base model. When receiving an evaluation request for the "visa" scenario, the system quickly loads the visa service scenario-specific adaptation module and combines it with the base model to form a candidate scenario expert model, efficiently completing the operational effect evaluation of the scenario. The integrated operational effect evaluation model only requires the deployment of one base model and four lightweight adaptation modules, reducing deployment resources by 70% compared to traditional multi-model solutions.

[0085] This embodiment first reduces memory usage by quantizing the parameters of a single base model, then trains a scenario-specific adaptation module on a designated projection layer for each business scenario, optimizes the training process by combining gradient accumulation and dynamic sparsity mechanism of model parameters, and finally integrates all adaptation modules and base model to form an operational performance evaluation model. This effectively solves the problems of severe interference between scenarios, high resource consumption, and complex deployment of traditional multi-scenario models. By using S-LoRA technology, a single base model can be adapted to multiple scenarios. With the help of various optimization mechanisms, the training and deployment costs are greatly reduced, while ensuring the accuracy and specificity of operational performance evaluation for each scenario.

[0086] In one feasible implementation, the dynamic sparsity mechanism for setting model parameters includes: Step S51, in the initial training stage, the weight gradient of the specified projection layer is globally sampled, and the weights with gradient contribution below the preset contribution threshold are subjected to initial sparsification. It should be noted that the initial training phase refers to the initial stage after the scene-specific adaptation module training starts. At this stage, stable weight parameters have not yet been formed, and sparsification is needed to select core weights. The specified projection layer refers to the network layer used for vector projection in a single pedestal model, specifically the q_proj, k_proj, v_proj, and o_proj layers, which are the core target layers for LoRA fine-tuning. Weight gradient refers to the gradient change value of the weight parameters during model training, reflecting the degree of influence of the weights on the model training effect. Global sampling refers to the comprehensive collection of gradients of all weights in the specified projection layer, without omitting any weight gradient information. Gradient contribution refers to the degree of contribution of the weight gradient to the model's adaptation to the scene training task, quantified by the magnitude of the absolute gradient value. The preset contribution threshold is a pre-set critical value used to determine whether weights need sparsification; weights below this threshold are considered low-contribution weights. Initial sparsification refers to setting low-contribution weights to 0 and freezing their gradients during the initial training phase, temporarily preventing them from participating in model training updates.

[0087] The purpose of this step is to quickly remove weights that contribute very little to scene adaptation during the early stages of training, thereby reducing model memory usage. Simultaneously, it focuses on training the core weights, improving training efficiency and laying the foundation for subsequent dynamic sparsity adjustments. In one possible implementation, the preset contribution threshold can be dynamically adjusted based on the complexity of the scene training data and the model performance requirements. When the data complexity is high, the threshold can be appropriately lowered to retain more weights for initial training.

[0088] Specifically, after the scene-specific adaptation module training starts, the system enters the initial training phase. It globally samples all weight gradients of the specified projection layers (q_proj, k_proj, v_proj, o_proj layers) of a single base model to obtain the absolute value of the gradient of each weight. The absolute value of the gradient of each weight is compared with a preset contribution threshold, and weights with gradient contributions lower than the threshold are selected. Initial sparsity processing is performed on these low-contribution weights, setting the weight value to 0 and freezing their gradient updates so that they do not change parameters temporarily in subsequent training. The position and number of the initially sparsified weights are recorded to form an initial sparsity mask, which provides a reference for subsequent sparsity range updates.

[0089] For example, in training a dedicated adaptation module for the "hotel booking scenario", the system globally samples the gradients of all weights in the q_proj, k_proj, v_proj, and o_proj layers during the initial training phase. A contribution threshold of 0.05 is preset, and weights with absolute gradient values ​​less than 0.05 are selected, accounting for 20% of the total weights. These weights undergo initial sparsity processing by setting their weight values ​​to 0 and freezing their gradients. An initial sparse mask is formed, recording the position information of these weights, thus completing the sparsity operation in the initial training phase.

[0090] Step S52: According to the preset training step size period, recalculate the gradient change magnitude of each weight in the specified projection layer, and update the current sparsified weight range according to the gradient change magnitude. Among them, the weights whose gradient contribution is still lower than the preset contribution threshold are retained in the sparsified state, and newly added weights whose gradient contribution is lower than the preset contribution threshold are added to the sparsified range. It should be noted that the preset training step size period refers to the pre-set interval of training steps used to update the sparsity weight range. In this embodiment, it is set to 100 training steps per period. The gradient change magnitude refers to the magnitude of the gradient change of each weight in the specified projection layer within one training step size period, which can reflect the change in the contribution of the weight to the model training within that period. The current sparsity weight range refers to the set of weights currently in a sparsity state, consisting of the initial sparsification process and the sparse weights updated in previous periods. Retaining the sparsity state means that for the original sparse weights whose gradient contribution is still lower than the preset threshold, their weights will continue to be kept at 0 and their gradients frozen. The newly added weights with a gradient contribution lower than the preset contribution threshold refer to the original non-sparse weights whose gradient contribution has dropped below the threshold within the current training step size period.

[0091] The core objective of this step is to dynamically adjust the sparsity weight range, promptly incorporating weights that become low-contribution during training into the sparsity range, continuously optimizing the model parameter structure, further reducing memory usage, and ensuring that model training resources are concentrated on high-contribution weights, thereby improving training efficiency and model performance. In one possible implementation, the preset training step size period can be flexibly adjusted according to the convergence speed of the model training; when the convergence speed is slow, the period can be shortened, and the sparsity range can be updated more frequently.

[0092] Specifically, the system, according to a preset training step size period (every 100 training steps), recalculates the gradient change magnitude of all weights in the specified projection layer at the end of each period; for weights currently in a sparse state, it checks whether their gradient contribution is still lower than a preset contribution threshold, and if it is still lower than the threshold, it retains their sparse state; for weights currently not sparse, it determines whether their gradient contribution has dropped below the preset contribution threshold, and filters out newly added low contribution weights; it includes the newly added low contribution weights in the sparse range and performs sparse processing (reset weights to 0, freeze gradients); it updates the sparse mask, records the position information of all currently sparse weights, and completes the dynamic update of the sparse weight range.

[0093] For example, the preset training step size for the "Visa Service Scenario" dedicated adaptation module is 100 training steps. At the end of the first cycle, the system recalculates the gradient change magnitude of each weight in the specified projection layer. Of the original 20% sparse weights, 90% of the weight gradient contribution is still below the threshold of 0.05, and the sparsity state is retained. At the same time, it is found that 10% of the original non-sparse weights have dropped below 0.05 in gradient contribution, becoming newly added low contribution weights. These newly added weights are included in the sparsity range, and the weight reset to 0 and gradient freeze operations are performed. The sparse mask is updated, and at this time the sparse weights account for 28% of the total weights, completing the update of the sparse weight range.

[0094] Step S53: For weights that are already in a sparse state, continuously monitor the gradient change of the weights. When the gradient contribution of the weights is detected to rise to the preset activation standard, de-sparse state of the weights and reactivate them so that the weights can participate in the model training process.

[0095] It should be noted that continuous monitoring refers to real-time tracking and detection of gradient changes in weights already in a sparse state at each training step, ensuring timely capture of changes in weight contribution. Increased gradient contribution means that the absolute value of the gradient of a sparse weight increases during training, indicating an increased contribution value of that weight to the current scene adaptation. The preset activation criterion refers to a pre-defined critical condition for determining whether a sparse weight needs to be reactivated; in this embodiment, it is set to 1.5 times the preset contribution threshold. De-sparsening means canceling the weight reset to 0 and gradient freeze operations on activated weights, restoring their normal parameter update permissions. Reactivation means allowing weights that have been de-sparsed to participate in model training again, and their weights can be updated and adjusted according to gradient changes.

[0096] The purpose of this step is to avoid misjudging high-potential weights due to initial or periodic sparsity processing. Through continuous monitoring and dynamic activation mechanisms, weights with subsequently increasing contributions are re-involved in training, ensuring that the model can fully utilize all valuable weight parameters, balancing memory optimization and model performance, and avoiding underfitting. In one possible implementation, the preset activation criterion can be set to the top 10% of all weights in terms of gradient contribution ranking. By using relative ranking rather than a fixed threshold, the need for weight reactivation can be determined more flexibly.

[0097] Specifically, in each training step, the system continuously monitors the weights that are already in a sparse state, and obtains the gradient changes of these weights in real time. The monitored gradient contribution of the weights is compared with the preset activation criteria to determine whether the activation conditions are met. When the gradient contribution of a sparse weight rises to the preset activation criteria, the system removes the sparsity of the weight, restores its updateability, and cancels the gradient freeze. The weight is removed from the sparse mask so that it can re-participate in the subsequent model training process, and the weight parameters can be updated and adjusted normally according to the gradient changes. The position and activation time of the reactivated weight are recorded to provide data support for subsequent training effect analysis.

[0098] For example, in the training of the dedicated adaptation module for "utility payment scenarios", the system continuously monitors the sparsed weights; the preset activation criterion is that the gradient contribution increases to 1.5 times the preset contribution threshold (0.05), i.e., 0.075; at the 300th training step, it is detected that the gradient contribution of a certain part of the sparsed weights reaches 0.08, which meets the activation criterion; the system removes the sparse state of these weights, restores their weight update permissions, and removes them from the sparse mask; these weights re-participate in model training, adjust parameters according to gradient changes, and improve the model's adaptability to utility payment scenarios.

[0099] This embodiment quickly eliminates low-contribution weights through initial sparsification during the initial training phase; dynamically updates the sparsified weight range according to a preset training step size cycle, continuously incorporating new low-contribution weights; and continuously monitors and dynamically activates high-contribution weights after sparsification. This effectively solves the problems of high memory consumption and low training efficiency in traditional model training. Through precise screening and dynamic adjustment, it significantly reduces memory consumption while ensuring that the model focuses on training core weights, thus balancing training efficiency and model performance.

[0100] In one feasible implementation, when a new scenario is identified, a target scenario expert model that is relevant to the new scenario in terms of scenario attributes or business characteristics is selected from the candidate scenario expert models, and transfer learning is performed based on the target scenario expert model to complete the adaptation to the new scenario, including: Step S61: Identify the scene attributes and business characteristics of the new scene; based on the scene attributes and business characteristics, select target scene expert models whose relevance meets the preset relevance threshold from each candidate scene expert model. It should be noted that candidate scenario expert models refer to model instances that have been trained and adapted to various existing business scenarios, consisting of a single base model and a corresponding scenario-specific adaptation module. The preset relevance threshold refers to a pre-set critical value used to determine whether the candidate scenario expert model meets the relevance standard for the new scenario; models exceeding this threshold can be used as target scenario expert models. Target scenario expert models refer to trained models selected from the candidate models that are highly relevant to the scenario attributes and business characteristics of the new scenario; they are the foundational models for transfer learning. The core purpose of this step is to find a suitable knowledge transfer carrier for the new scenario. By selecting highly relevant candidate models, it ensures the effective reuse of model features and knowledge in the subsequent transfer learning process, solves the problem of poor training performance caused by data sparsity in the new scenario, and lays the foundation for rapid adaptation to the new scenario.

[0101] In one possible implementation, the calculation of scene relevance can adopt a multi-dimensional weighted scoring mechanism, which scores from multiple dimensions such as scene attribute similarity, business feature similarity, and data distribution similarity, and then compares the weighted sum with a preset relevance threshold to improve the accuracy of the screening.

[0102] Specifically, the system obtains basic information about the new scenario through the business configuration interface. Based on preset attribute and feature extraction rules, it automatically identifies the scenario attributes (such as industry category, service type, target user group, etc.) and business characteristics (such as recommendation strategy, core indicators, interaction process, etc.) of the new scenario. It extracts the scenario attributes and business characteristics corresponding to each candidate scenario expert model and constructs a candidate model feature library. It uses the cosine similarity algorithm to calculate the similarity between the new scenario and each candidate model in terms of scenario attributes and business characteristics. It then combines preset weights to perform a weighted calculation of the two similarities to obtain the comprehensive relevance score between the new scenario and each candidate model. The comprehensive relevance score of each candidate model is compared with a preset relevance threshold. Candidate models with comprehensive relevance scores higher than the threshold are selected. If there are multiple models that meet the conditions, the model with the highest score is selected as the target scenario expert model.

[0103] For example, the new scenario is "Life Payment Scenario". The system identifies its scenario attributes as "Life service industry, payment service, target users are all citizens, local service scope", and its business characteristics as "recommendation strategy focuses on convenience, core indicators are payment success rate and user activity, simple and direct interaction process, and goal is to improve user payment efficiency". The system extracts the features of "Convenience Service Scenario" from the candidate scenario expert model and calculates that the similarity between the two scenario attributes is 0.85, the similarity between business features is 0.82, and the weighted comprehensive score is 0.84. The preset relevance threshold is 0.7. This score is higher than the threshold and is the highest among all candidate models. Therefore, the "Convenience Service Scenario" expert model is determined as the target scenario expert model.

[0104] Step S62: Load the weight parameters of the target scene expert model, freeze the specified projection layer parameters responsible for the attention mechanism in the weight parameters, and retain the general features learned by the model. It should be noted that the weight parameters of the target scene expert model refer to all parameter values ​​learned by the target model during training, including the basic parameters of the single base model and the training parameters of the scene-specific adaptation module (LoRA Adapter). These parameters carry the features and knowledge learned by the model in the original scene. The specified projection layer refers to the network layer in the model responsible for the attention mechanism. In this embodiment, it is specifically the q_proj layer and the k_proj layer. The q_proj layer is responsible for projecting the query vector, and the k_proj layer is responsible for projecting the key vector. Both of them jointly affect the attention structure of the model.

[0105] Furthermore, freezing refers to preventing the parameters of a specified projection layer from being updated during subsequent fine-tuning, thus maintaining their parameter values ​​unchanged. In this embodiment, only the q_proj and k_proj layers of LoRA are frozen, and only the v_proj and o_proj layers are trained and fine-tuned. The q_proj and k_proj layers are responsible for the projection of query and key vectors, mainly affecting the model's attention mechanism. Freezing them preserves the original model's attention structure, which helps maintain the model's stability. The v_proj and o_proj layers are mainly used to process value vectors and output projections, and are important links in the model's output generation in specific tasks. They are highly sensitive to knowledge of specific tasks, and training them helps the model understand and adapt to the requirements of specific tasks. The attention mechanism refers to the core mechanism used by the model to focus on key information and capture data relationships, which helps the model understand the importance and internal connections of input data. General features refer to the common features learned by the target scene expert model during training in the original scene that are applicable to multiple scenarios, such as user behavior analysis logic, data indicator interpretation methods, and general language understanding capabilities.

[0106] The purpose of this step is to preserve the general features and stable attention structure already learned by the target model, avoid over-covering or destroying this valuable knowledge during fine-tuning in new scenarios, and at the same time reduce the parameter scale of fine-tuning in new scenarios by freezing some parameters, thereby reducing training costs and the risk of overfitting, and ensuring the stability of the model and the efficiency of knowledge reuse during the transfer learning process.

[0107] In one possible implementation, the specified projection layers that are frozen can be flexibly adjusted according to the correlation between the new scene and the original scene of the target model. If the correlation between the two is extremely high, more layers of parameters can be frozen; if the correlation is moderate, the number of frozen layers can be appropriately reduced to improve the model's adaptability to the new scene.

[0108] Specifically, the system reads the weight parameter file of the expert model for the target scene through the model loading interface, loads all parameters into the training framework, and completes model initialization. Based on the preset frozen layer configuration (q_proj layer and k_proj layer), it iterates through all network layer parameters of the model, identifies and locks the parameters of q_proj layer and k_proj layer, and sets their gradient update switch to the off state so that the parameters remain unchanged during subsequent fine-tuning. The frozen model is then parameter-verified to confirm that the parameters of q_proj layer and k_proj layer do not have update permissions, while the parameters of other layers can be updated normally. The system retains the general features already learned by the model (such as user behavior analysis logic, general language understanding ability, etc.) to provide a foundation for learning new scene-specific features in the future.

[0109] For example, the target scenario expert model is the "convenience service scenario" expert model. The system loads the weight parameters of this model, including the basic parameters of the Qwen2.5-3B-Instruct base model and the corresponding LoRA Adapter parameters. Through the parameter configuration interface of the training framework, the parameters of the q_proj layer and k_proj layer in the model are frozen, and their gradient update function is turned off. It is verified that the parameters of these two layers cannot be modified in subsequent training, while the parameters of the v_proj layer, o_proj layer and other unfrozen layers can be updated normally. The model retains the general features learned in the "convenience service scenario", such as "service evaluation logic centered on user convenience" and "simple and direct user interaction data analysis method", providing knowledge support for the adaptation of the "life payment scenario".

[0110] Step S63: Adjust the training learning rate. Based on the scene training data corresponding to the new scene, fine-tune the target scene expert model through model feature sharing and knowledge transfer to form a scene expert model corresponding to the new scene, thus completing the new scene adaptation.

[0111] It's important to note that the training learning rate refers to the step size of parameter updates during model training. It determines the speed and stability of model learning; an excessively high learning rate can lead to model oscillations and non-convergence, while an excessively low rate will prolong the training cycle. Model feature sharing refers to the reuse of general features from the target scenario's expert model during fine-tuning in the new scenario, providing a foundation for feature learning in the new scenario. Knowledge transfer refers to transferring business knowledge and data processing logic learned by the target model in the original scenario to the new scenario, helping the new scenario model quickly master core capabilities. Fine-tuning refers to making small updates to the trainable layer parameters of the target model based on the new scenario's training data, while freezing some parameters. This allows the model to retain general features while learning features specific to the new scenario. The scenario expert model corresponding to the new scenario refers to a model instance that, after fine-tuning, can accurately adapt to the operational performance evaluation requirements of the new scenario. New scenario adaptation refers to completing the training and deployment of the new scenario expert model, enabling the model to effectively handle operational performance evaluation requests in the new scenario.

[0112] The purpose of this step is to transfer the general features and knowledge of the target model to new scenarios by adjusting the learning rate and fine-tuning the model based on a small amount of training data. This allows for the rapid construction of an expert model adapted to the new scenario, addressing the issues of cold start and data sparsity in new scenarios. Simultaneously, it reduces training costs, shortens the training cycle, and ensures the quality of operational performance evaluation in the new scenario. In one possible implementation, the learning rate can be adjusted dynamically. A lower initial learning rate is used in the early stages of training, gradually increasing it with each training iteration, and then decreasing it once the loss value stabilizes, balancing training stability and efficiency.

[0113] Specifically, the system determines the target learning rate for fine-tuning the new scenario based on the original learning rate of the target scenario expert model. The target learning rate is set to 1 / 4 of the original learning rate to avoid overfitting due to an excessively high learning rate. The system acquires the scenario training data corresponding to the new scenario, preprocesses the data according to a preset batch size (e.g., batch_size=4), and converts it into an input format recognizable by the model. The preprocessed new scenario training data is input into the target scenario expert model with some parameters frozen, and fine-tuning training begins. During training, the model shares the common features of the target model (e.g., user behavior analysis logic, general language understanding ability, etc.) and reuses the business knowledge of the original scenario through knowledge transfer. The model only updates the parameters of the unfrozen layers (e.g., v_proj layer, o_proj layer), gradually learning the specific features of the new scenario (e.g., payment success rate assessment, bill query data analysis, etc. in the life payment scenario). Through iterative training, the model's loss value on the new scenario training data converges to a preset range (e.g., loss≤2.5), and training stops. The trained model parameters are then solidified to form the scenario expert model corresponding to the new scenario, completing the new scenario adaptation and deployment to the operational performance evaluation system.

[0114] For example, the initial learning rate of the expert model for the target scenario "Convenience Service Scenario" is 1e-4. The system sets the fine-tuning target learning rate for the new scenario "Life Payment Scenario" to 2.5e-5. A small amount of training data for this new scenario (including 500 payment behavior statistics, scenario background information, etc.) is obtained, preprocessed, and input into the model at batch_size=4. During the fine-tuning process, the model retains the general features of the "Convenience Service Scenario," reuses the "User Convenience Evaluation Logic" through knowledge transfer, and updates the parameters of the v_proj layer and o_proj layer to learn the specific features of the "Life Payment Scenario," such as "Correlation Analysis between Payment Success Rate and User Activity" and "Summary of User Behavior Differences for Different Payment Types." After 200 training steps, the model loss value converges to 2.3, meeting the preset requirements, and training stops. The model parameters are solidified to form the "Life Payment Scenario" expert model. After deployment, it can accurately handle the operational effect evaluation requests for this scenario, completing the adaptation to the new scenario.

[0115] This embodiment first identifies the scene attributes and business characteristics of the new scenario, then selects a target scene expert model with high relevance, loads the target model weights and freezes the parameters of the specified projection layer responsible for the attention mechanism, and finally adjusts the learning rate and fine-tunes it based on a small amount of training data from the new scenario. This effectively solves the problems of poor model training effect, long cycle, and high cost caused by cold start and data sparsity in new scenarios. By reusing the general features and knowledge of relevant scene expert models, only a small number of parameters need to be fine-tuned to quickly adapt to new scenarios, which greatly shortens the training cycle of new scenario models, reduces training costs, and ensures the accuracy and stability of the evaluation of the operational effect of new scenarios. It provides efficient technical support for the rapid expansion of multi-scenario businesses.

[0116] In one feasible implementation, adjusting the training learning rate involves fine-tuning the target scene expert model based on the scene training data corresponding to the new scene, through model feature sharing and knowledge transfer, to form a scene expert model corresponding to the new scene, thus completing the new scene adaptation. This includes: Step S71: Determine the target learning rate for fine-tuning the new scene based on the original learning rate of the target scene expert model, wherein the target learning rate is lower than the original learning rate; It's important to clarify that the original learning rate refers to the learning rate used by the expert model in the target scenario during its training in the corresponding original scenario. This learning rate is optimized for the training data and business features of the original scenario, ensuring efficient convergence of the model in the original scenario. The target learning rate, on the other hand, refers to the learning rate used during fine-tuning in the new scenario, and its value is lower than the original learning rate. The core purpose of this step is to avoid excessively large model parameter updates due to an excessively high learning rate when the amount of training data in the new scenario is small and features are not fully explored. This would damage the general features and knowledge already learned by the target model, while also preventing overfitting on small sample data. This ensures a stable and efficient fine-tuning process, laying the foundation for the model to quickly adapt to the new scenario.

[0117] In one possible implementation, the ratio of the target learning rate to the original learning rate can be dynamically adjusted according to the scale of the training data for the new scenario. The smaller the amount of data, the lower the ratio should be. For example, when the amount of data is less than 1,000, the target learning rate can be set to 1 / 5 of the original learning rate; when the amount of data is between 1,000 and 5,000, it can be set to 1 / 4.

[0118] Specifically, the system first retrieves the training configuration file of the expert model for the target scene and extracts the original learning rate value used during training in the original scene. Based on the scale of the training data for the new scene and the correlation between the new scene and the original scene, the ratio of the target learning rate to the original learning rate is determined. In this embodiment, the default ratio is 1 / 4. The target learning rate for fine-tuning the new scene is calculated by multiplying the original learning rate by this ratio. The target learning rate is validated to ensure that it is within a reasonable learning rate range for model training (e.g., between 1e-5 and 1e-4). After the validation is passed, it is used as the learning rate parameter for fine-tuning the new scene.

[0119] For example, the expert model for the target scenario is the "convenience service scenario" expert model, and its original learning rate is 1e-4; the new scenario is the "utility payment scenario", which has 800 training data points and is highly correlated with the "convenience service scenario". The ratio of the target learning rate to the original learning rate is determined to be 1 / 4; the calculated target learning rate is 2.5e-5, and the value is verified to be within a reasonable range. It is then used as the learning rate parameter for fine-tuning the "utility payment scenario".

[0120] Step S72: Input the scene training data corresponding to the new scene into the target scene expert model in a preset batch. During the fine-tuning process, keep the frozen layer parameters unchanged and only update the trainable layer parameters. It's important to note that the training data for the new scenario refers to the adaptive training data generated specifically for the new scenario. This data includes background information, some user behavior statistics, and business objectives, serving as the core basis for the model to learn the unique features of the new scenario. The preset batch size refers to the pre-defined amount of data input for a single training iteration (batch_size), determined based on the model's memory usage and training efficiency requirements. The target scenario expert model refers to a model with weights loaded and some layer parameters frozen. The frozen layers are the q_proj and k_proj layers responsible for the attention mechanism, while the trainable layers are the v_proj, o_proj, and other unfrozen network layers. "Freezing layer parameters unchanged" means that during fine-tuning, the weight parameters of the q_proj and k_proj layers remain at their initial values ​​without any updates, preserving the model's general features and attention structure. "Updating only trainable layer parameters" means that during model training, only the weight parameters of trainable layers such as the v_proj and o_proj layers are adjusted, allowing these layers to gradually learn the unique features of the new scenario. The purpose of this step is to update the parameters of the trainable layers in a targeted manner using limited new scene training data, while protecting the existing knowledge and structural stability of the target model. This enables the model to accurately learn the features of the new scene, and at the same time, it reduces the memory consumption of a single training session and improves training efficiency by using batch input.

[0121] In one possible implementation, the preset batch can be dynamically adjusted according to the model quantization accuracy and the size of the video memory. For example, the preset batch can be set to 8 for a 4-bit quantized model and 4 for an 8-bit quantized model, to ensure that there is no video memory overflow problem during the training process.

[0122] Specifically, the system acquires the scene training data corresponding to the new scene, divides the data into multiple training batches according to a preset batch size (4 in this embodiment); preprocesses the data of the first training batch, converting it into a tensor format that the model can recognize; inputs the preprocessed training batch data into the target scene expert model and starts fine-tuning training; during the training process, the parameter update switch of the frozen layer (q_proj layer, k_proj layer) is locked through the model training framework to ensure that its parameters remain unchanged; only the parameters of the trainable layer (v_proj layer, o_proj layer) are allowed to be calculated and updated according to the training loss value; all training batches are input into the model in sequence, and the above process is repeated until the training iteration of all batches is completed.

[0123] Step S73: Through iterative training, the loss value of the target scene expert model on the new scene training data is brought to a preset range. With the help of model feature sharing and knowledge transfer, the model learns the special features of the new scene and completes the adaptation to the new scene.

[0124] It should be noted that iterative training refers to the process of repeatedly inputting all training batches of data into the model for training. Each round of input and parameter update for all batches constitutes one iteration, and the model parameters are gradually optimized through multiple iterations. The loss value is a quantitative indicator of the error between the model's output and the labels of the training data in the new scene. It reflects the model's learning effect on the features of the new scene; a lower loss value indicates a better model adaptation. The preset range refers to a pre-defined range of loss values ​​used to determine whether model training is complete; in this embodiment, it is set to loss ≤ 2.5. Model feature sharing refers to the reuse of general features (such as user behavior analysis logic and general language understanding ability) learned by the target scene expert model in the original scene during the fine-tuning process in the new scene, providing a foundation for learning new scene features. Knowledge transfer refers to the transfer of business knowledge and data processing rules accumulated by the target model in the original scene to the new scene, helping the model quickly understand the business logic and core requirements of the new scene.

[0125] Furthermore, new scenario-specific features refer to the unique characteristics of a new scenario, such as data distribution, business objectives, and text style. Examples include the payment success rate analysis and user behavior characteristics for bill inquiries in the "utility bill payment scenario." New scenario adaptation refers to the model's ability, through iterative training, to retain both general features and transferred knowledge while mastering new scenario-specific features. This enables the model to accurately output an evaluation summary of the operational effectiveness of the new scenario, thus achieving model adaptation to the new scenario. The purpose of this step is to allow the model to gradually optimize its parameters through multiple rounds of iterative training, bringing the loss value to a preset range. This ensures the model fully learns the new scenario-specific features and, through feature sharing and knowledge transfer, achieves high-quality new scenario adaptation even with small sample data, solving the cold start problem in new scenarios.

[0126] In one possible implementation, the preset range can be flexibly adjusted according to the business accuracy requirements of the new scenario. For scenarios with high accuracy requirements (such as financial scenarios), the preset range can be set to loss ≤ 2.0; for scenarios with moderate accuracy requirements (such as life service scenarios), it can be set to loss ≤ 2.5. Additionally, it should be noted that in this embodiment, the termination conditions for iterative training include two situations: first, the loss value converges to the preset range and there is no significant increase after three consecutive iterations; second, the number of iterations reaches the preset maximum number of iterations (such as 500 rounds), and training stops even if the loss value has not fully converged, to avoid overfitting due to overtraining.

[0127] Specifically, the system initiates multiple rounds of iterative training. In each round, all training batches of data are input sequentially, and the parameters of the trainable layers are updated. After each round of iteration, the loss value of the model on the training data of the new scenario is calculated. The current loss value is compared with a preset range to determine whether the convergence condition is met. If the loss value has not converged and the maximum number of iterations has not been reached, the next round of iterative training continues. If the loss value converges to the preset range and remains stable for three consecutive rounds, or the maximum number of iterations is reached, the iterative training stops. During the training process, the model reuses the general features of the target model through feature sharing, absorbs the business knowledge of the original scenario through knowledge transfer, and gradually masters the features specific to the new scenario. After training stops, the current parameters of the model are fixed to form a scenario expert model corresponding to the new scenario, thus completing the adaptation to the new scenario.

[0128] For example, the preset loss value range for the new scenario "utility bill payment scenario" is loss ≤ 2.5, and the maximum number of iterations is 500 rounds. When the model is trained to the 200th round, the model loss value drops to 2.3, and the loss values ​​for the subsequent 3 rounds are 2.28, 2.25, and 2.23, respectively, all within the preset range and steadily decreasing. The system determines that the model has converged and stops iterative training. At this point, the model retains the general features of the "convenience service scenario" (such as the user convenience assessment logic), and has also mastered the processing rules of similar businesses through knowledge transfer. At the same time, it has learned the specific features of the "utility bill payment scenario" (such as the correlation analysis between payment success rate and user activity, and the differences in user behavior for different payment types). The model parameters are solidified to form an expert model for the "utility bill payment scenario", completing the adaptation to the new scenario.

[0129] This embodiment first determines a lower target learning rate based on the original learning rate of the expert model for the target scenario. Then, it inputs new scenario training data in preset batches, keeping the parameters of the frozen layer unchanged and only updating the parameters of the trainable layer. Finally, it iterates through training to bring the loss value to a preset range. By leveraging feature sharing and knowledge transfer, the model learns the unique features of the new scenario, thus effectively solving the problems of sparse training data and cold start difficulties in new scenarios. The low learning rate ensures training stability, batch training reduces memory consumption, and multiple iterations achieve accurate model adaptation. At the same time, it fully reuses the existing knowledge and features of the target model, significantly reducing the training cost and cycle of the new scenario model. This ensures the accuracy and stability of the evaluation of the operational effect of the new scenario and provides strong support for the rapid expansion of multi-scenario businesses.

[0130] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0131] This application also provides an operational performance evaluation device, please refer to... Figure 3The operational performance evaluation device includes: The acquisition module 31 is used to acquire the target scenario and evaluation time interval configured by the user. Evaluation module 32 is used to input the target scenario and the evaluation time interval into the operation effect evaluation model to obtain the operation effect evaluation summary output by the operation effect evaluation model. The operation effect evaluation model consists of a single base model and multiple scenario-specific adaptation modules. The operation effect evaluation model is obtained by performing multi-scenario-specific fine-tuning training on scenario training data.

[0132] The operational performance evaluation device provided in this application, employing the operational performance evaluation method in the above embodiments, can solve the technical problems in the background art. Compared with the prior art, the beneficial effects of the operational performance evaluation device provided in this application are the same as the beneficial effects of the operational performance evaluation method provided in the above embodiments, and other technical features in the operational performance evaluation device are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0133] This application provides an operational performance evaluation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the operational performance evaluation method in Embodiment 1 above.

[0134] The following is for reference. Figure 4 The diagram illustrates a structural schematic of an operational performance evaluation device suitable for implementing embodiments of this application. The operational performance evaluation device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The operational effectiveness evaluation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0135] like Figure 4As shown, the operational performance evaluation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the operational performance evaluation device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the operational effectiveness evaluation equipment to communicate wirelessly or wiredly with other equipment to exchange data. While the figure shows operational effectiveness evaluation equipment with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0136] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0137] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0138] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the operational performance evaluation methods provided by the methods described above.

[0139] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0141] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for evaluating operational effectiveness, characterized in that, include: Obtain the target scenario and evaluation time range configured by the user; The target scenario and the evaluation time interval are input into the operation effect evaluation model to obtain the operation effect evaluation summary output by the operation effect evaluation model. The operation effect evaluation model consists of a single base model and multiple scenario-specific adaptation modules. The operation effect evaluation model is obtained by performing multi-scenario-specific fine-tuning training on scenario training data.

2. The operational performance evaluation method as described in claim 1, characterized in that, Before inputting the target scenario and the evaluation time interval into the operational performance evaluation model to obtain the operational performance evaluation summary output by the operational performance evaluation model, the method further includes: Construct scene training data that is adaptable to multiple scenarios; Based on the training data of the aforementioned scenarios, the single base model is subjected to multi-scenario-specific fine-tuning training. The training process is optimized by integrating model quantization, gradient accumulation, and dynamic sparsity mechanism of model parameters, forming an operation effect evaluation model equipped with multiple scenario-specific adaptation modules. The operation effect evaluation model includes the candidate scenario expert models that have been trained for each scenario. When a new scenario is identified, a target scenario expert model that is related to the new scenario in terms of scenario attributes or business characteristics is selected from the candidate scenario expert models, and transfer learning is performed based on the target scenario expert model to complete the adaptation of the new scenario.

3. The operational performance evaluation method as described in claim 2, characterized in that, The construction of scene training data adapted to multiple scenarios includes: Obtain the contextual information and historical user behavior data corresponding to each business scenario; A scene background knowledge base is constructed based on the scene background information, and the historical user behavior data is converted into vector form and stored to construct a behavior vector library. Obtain daily user behavior statistics for each of the aforementioned business scenarios, and retrieve relevant historical behavior data from the behavior vector library based on the daily user behavior statistics. By using retrieval-enhanced generation technology, the retrieved relevant historical behavior data and the corresponding scene background information in the scene background knowledge base are input into a general large model to generate scene training data adapted to each of the aforementioned business scenarios.

4. The operational performance evaluation method as described in claim 2, characterized in that, Based on the training data of the aforementioned scenarios, the single base model undergoes multi-scenario-specific fine-tuning training. This involves integrating model quantization, gradient accumulation, and dynamic sparsity mechanisms for model parameters to optimize the training process, resulting in an operational performance evaluation model equipped with multiple scenario-specific adaptation modules. This includes: The single base model is subjected to parameter quantization processing; For any business scenario, based on the scenario training data corresponding to the business scenario, a scenario-specific adaptation module corresponding to the business scenario is trained on a specified projection layer of the single base model. During the training process, the training batch is split by gradient accumulation and a dynamic sparsity mechanism for model parameters is set to sparsify the weights with low contribution in the model and reactivate the sparsified weights that meet the activation conditions. After training of all scenario-specific adaptation modules is completed, each scenario-specific adaptation module is integrated with a single base model to form an operational performance evaluation model that includes candidate scenario expert models corresponding to each business scenario.

5. The operational performance evaluation method as described in claim 4, characterized in that, The dynamic sparsity mechanism for setting model parameters includes: In the initial training phase, the weight gradient of the specified projection layer is globally sampled, and the weights with gradient contributions lower than a preset contribution threshold are subjected to initial sparsification. According to the preset training step size period, the gradient change magnitude of each weight in the specified projection layer is recalculated, and the current sparsified weight range is updated according to the gradient change magnitude. Among them, the weights whose gradient contribution is still lower than the preset contribution threshold are retained in the sparsified state, and newly added weights whose gradient contribution is lower than the preset contribution threshold are added to the sparsified range. For weights that are already in a sparse state, the gradient change of the weights is continuously monitored. When the gradient contribution of the weights is detected to rise to the preset activation standard, the sparse state of the weights is removed and they are reactivated so that the weights can participate in the model training process.

6. The operational performance evaluation method as described in claim 2, characterized in that, When a new scenario is identified, a target scenario expert model that is relevant to the new scenario in terms of scenario attributes or business characteristics is selected from the candidate scenario expert models, and transfer learning is performed based on the target scenario expert model to complete the adaptation to the new scenario, including: Identify the scenario attributes and business characteristics of the new scenario, and based on the scenario attributes and business characteristics, select target scenario expert models whose relevance meets a preset relevance threshold from each candidate scenario expert model; Load the weight parameters of the expert model for the target scene, freeze the specified projection layer parameters responsible for the attention mechanism in the weight parameters, and retain the general features already learned by the model; The training learning rate is adjusted, and based on the scene training data corresponding to the new scene, the target scene expert model is fine-tuned through model feature sharing and knowledge transfer to form a scene expert model corresponding to the new scene, thus completing the adaptation to the new scene.

7. The operational performance evaluation method as described in claim 6, characterized in that, The adjustment of the training learning rate, based on the scene training data corresponding to the new scene, involves fine-tuning the target scene expert model through model feature sharing and knowledge transfer to form a scene expert model corresponding to the new scene, thus completing the adaptation to the new scene. This includes: The target learning rate for fine-tuning the new scene is determined based on the original learning rate of the expert model for the target scene, and the target learning rate is lower than the original learning rate. The scene training data corresponding to the new scene is input into the target scene expert model in a preset batch. During the fine-tuning process, the parameters of the frozen layer remain unchanged, and only the parameters of the trainable layer are updated. Through iterative training, the loss value of the target scene expert model on the new scene training data converges to a preset range. With the help of model feature sharing and knowledge transfer, the model learns the special features of the new scene and completes the adaptation to the new scene.

8. An operational performance evaluation device, characterized in that, include: The acquisition module is used to acquire the target scenario and evaluation time range configured by the user. The evaluation module is used to input the target scenario and the evaluation time interval into the operation effect evaluation model to obtain the operation effect evaluation summary output by the operation effect evaluation model. The operation effect evaluation model consists of a single base model and multiple scenario-specific adaptation modules. The operation effect evaluation model is obtained by performing multi-scenario-specific fine-tuning training on scenario training data.

9. An operational performance evaluation device, characterized in that, The operational performance evaluation device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the operational performance evaluation method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the operational performance evaluation method as described in any one of claims 1 to 7.

Images