Business agent evaluation method, device and equipment

Abstract

The embodiment of the invention discloses a service agent evaluation method, device and equipment. The method comprises the following steps: receiving an evaluation request for a target service agent; copying service flow data from a target service needing to be deployed by the target service agent, and performing service prediction processing through the target service agent based on the service flow data to obtain a corresponding service prediction result; determining competition situation simulation information corresponding to the target service through a preset dynamic game model based on historical service change trend information and the service prediction result in the target service by taking the target service agent as a dominator and a preset reference prediction model as a follower; and based on the competition situation simulation information corresponding to the target service and the service prediction result, through a preset service prediction model, evaluating a target service agent needing to be deployed in the target service to obtain a corresponding evaluation result.

Description

TECHNICAL FIELD

[0001] The present document relates to the technical field of computers, and particularly relates to a business intelligent agent evaluation method, device and equipment. BACKGROUND

[0002] In recent years, in the fields of e-commerce, leasing, second-hand transactions, etc., operational optimization algorithms play a crucial role, and new product forms such as intelligent hosting and intelligent pricing are constantly emerging. For example, in the case of commodity pricing decisions, intelligent models are used to dynamically determine the selling price (i.e., pricing) to help businesses fine-tune their operations, increase GMV, and reduce operational labor costs.

[0003] The quality of the intelligent agent directly affects the final results. To measure and optimize the performance of the intelligent model, a scientific and systematic evaluation can be performed. However, the usual evaluation methods have the disadvantages of data bias, limited indicators, or low efficiency and high cost. Therefore, a more optimal intelligent agent evaluation mechanism is needed to restore a certain length of trading environment for intelligent models (such as pricing models), hosting strategies, and recommendation algorithms within a short period of time (minutes to hours), to comprehensively test and evaluate the effectiveness of the algorithms. In a safe and controllable environment such as privacy protection, the effectiveness and risk of the intelligent agent can be quickly verified, which not only improves the accuracy and robustness of the model, but also significantly reduces the risk and cost of testing in a real environment. SUMMARY

[0004] The purpose of the embodiments of the present specification is to provide a more optimal intelligent agent evaluation mechanism to restore a certain length of trading environment for intelligent models (such as pricing models), hosting strategies, and recommendation algorithms within a short period of time (minutes to hours), to comprehensively test and evaluate the effectiveness of the algorithms. In a safe and controllable environment such as privacy protection, the effectiveness and risk of the intelligent agent can be quickly verified, which not only improves the accuracy and robustness of the model, but also significantly reduces the risk and cost of testing in a real environment.

[0005] To achieve the above technical solutions, the embodiments of the present specification are implemented as follows: The embodiment of the present specification provides a service intelligent agent evaluation method, which comprises the following steps: receiving an evaluation request for a target service intelligent agent, wherein the target service intelligent agent is used for preset service prediction processing of a target service required to be deployed by the target service intelligent agent; copying service traffic data from the target service required to be deployed by the target service intelligent agent, and performing service prediction processing through the target service intelligent agent based on the service traffic data to obtain a corresponding service prediction result; taking the target service intelligent agent as a leader and a preset benchmark prediction model as a follower, determining corresponding competition situation simulation information of the target service based on historical service change trend information in the target service and the service prediction result through a preset dynamic game model; and evaluating the target service intelligent agent required to be deployed in the target service through a preset service estimation model based on the corresponding competition situation simulation information of the target service and the service prediction result to obtain a corresponding evaluation result.

[0006] The embodiment of the present specification provides a service intelligent agent evaluation device, which comprises the following modules: an evaluation request module for receiving an evaluation request for a target service intelligent agent, wherein the target service intelligent agent is used for preset service prediction processing of a target service required to be deployed by the target service intelligent agent; a service prediction module for copying service traffic data from the target service required to be deployed by the target service intelligent agent, and performing service prediction processing through the target service intelligent agent based on the service traffic data to obtain a corresponding service prediction result; a competition simulation module for taking the target service intelligent agent as a leader and a preset benchmark prediction model as a follower, determining corresponding competition situation simulation information of the target service based on historical service change trend information in the target service and the service prediction result through a preset dynamic game model; and an intelligent agent evaluation module for evaluating the target service intelligent agent required to be deployed in the target service through a preset service estimation model based on the corresponding competition situation simulation information of the target service and the service prediction result to obtain a corresponding evaluation result.

[0007] An evaluation device for a business agent is provided, and the evaluation device for the business agent comprises a processor and a memory arranged to store computer executable instructions, which, when executed, cause the processor to: receive an evaluation request for a target business agent, the target business agent being configured to perform a preset business prediction process on a target business to which the target business agent is to be deployed; copy business traffic data from the target business to which the target business agent is to be deployed, and perform a business prediction process on the target business agent based on the business traffic data to obtain a corresponding business prediction result; take the target business agent as a leader and a preset benchmark prediction model as a follower, determine corresponding competitive situation simulation information of the target business based on historical business trend information of the target business and the business prediction result through a preset dynamic game model; and evaluate the target business agent to be deployed to the target business based on the corresponding competitive situation simulation information of the target business and the business prediction result through a preset business estimation model to obtain a corresponding evaluation result.

[0008] The embodiments of the present specification also provide a storage medium for storing computer executable instructions, which, when executed by a processor, implement the following processes: receiving an evaluation request for a target business agent, the target business agent being configured to perform a preset business prediction process on a target business to which the target business agent is to be deployed; copying business traffic data from the target business to which the target business agent is to be deployed, and performing a business prediction process on the target business agent based on the business traffic data to obtain a corresponding business prediction result; taking the target business agent as a leader and a preset benchmark prediction model as a follower, determining corresponding competitive situation simulation information of the target business based on historical business trend information of the target business and the business prediction result through a preset dynamic game model; and evaluating the target business agent to be deployed to the target business based on the corresponding competitive situation simulation information of the target business and the business prediction result through a preset business estimation model to obtain a corresponding evaluation result.

[0009] The embodiment of the present specification further provides a computer program product comprising a computer program which, when executed by a processor, implements the following process: receiving an evaluation request for a target business agent, the target business agent being used for performing a preset business prediction process on a target business which needs to be deployed by the target business agent; copying business traffic data from the target business which needs to be deployed by the target business agent, and performing a business prediction process through the target business agent based on the business traffic data to obtain a corresponding business prediction result; taking the target business agent as a leader and taking a preset benchmark prediction model as a follower, determining corresponding competitive situation simulation information of the target business through a preset dynamic game model based on historical business change trend information in the target business and the business prediction result; and evaluating the target business agent which needs to be deployed into the target business through a preset business estimation model based on the corresponding competitive situation simulation information of the target business and the business prediction result to obtain a corresponding evaluation result. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without paying creative labor; Figure 1 FIG. 1 is a structural schematic diagram of a business agent evaluation system according to an embodiment of the present specification; Figure 2 FIG. 2 is a schematic diagram of a business agent evaluation process according to an embodiment of the present specification; Figure 3 FIG. 3 is a schematic diagram of another business agent evaluation process according to an embodiment of the present specification; Figure 4 FIG. 4 is a schematic diagram of yet another business agent evaluation process according to an embodiment of the present specification; Figure 5 FIG. 5 is a schematic diagram of yet another business agent evaluation process according to an embodiment of the present specification; Figure 6 FIG. 6 is a schematic diagram of yet another business agent evaluation process according to an embodiment of the present specification; Figure 7 FIG. 7 is a schematic diagram of yet another business agent evaluation process according to an embodiment of the present specification; Figure 8 FIG. 8 is a schematic diagram of a processing process of a dynamic game model according to an embodiment of the present specification; Figure 9 FIG. 9 is a schematic diagram of a business estimation model according to an embodiment of the present specification; Figure 10A schematic diagram of an evaluation process of a business agent according to the present specification; Figure 11 A schematic diagram of an evaluation process of a business agent according to the present specification; Figure 12 A schematic diagram of an evaluation device of a business agent according to the present specification; Figure 13 A schematic diagram of an evaluation device of a business agent according to the present specification. DETAILED DESCRIPTION

[0011] The embodiments of the present specification provide a business agent evaluation method, device and equipment.

[0012] In order for those skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be described clearly and completely below with reference to the drawings in the embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, not all. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present specification.

[0013] The embodiments of the present specification provide an agent evaluation mechanism based on dynamic game. In recent years, in the fields of e-commerce, leasing, second-hand transactions, etc., operational optimization algorithms play a crucial role and new product forms such as intelligent hosting and intelligent pricing have also emerged. Taking commodity pricing decision as an example, by constructing an intelligent model to dynamically determine the selling price (i.e. pricing), it helps merchants to fine-tune operations to improve GMV and reduce operational labor costs. The quality of the agent directly affects the final result. Usually, the agent or model can be evaluated by offline evaluation or AB experiment evaluation. The offline evaluation is realized in the model screening and parameter tuning stage (i.e. before model deployment), that is, using historical data to simulate the test environment and verifying the model performance through preset indicators. However, there is data bias in this way. Specifically, historical data may not reflect the current data distribution (such as user interest drift), and there may also be indicator limitations, i.e. offline indicators (such as AUC) may be out of touch with business results. The AB experiment evaluation is in the verification stage before going online, which is to randomly divide the online traffic data into an experimental group (for a new model) and a control group (for an old model), and compare and analyze through real traffic data to get the corresponding evaluation results. However, this method needs to develop and configure the experimental scheme and deploy it online, so the cost is high, the test period is long, and the traffic is limited. In addition, this method is complex in engineering, and needs to ensure that the bucketing is unbiased (i.e. the data distribution of the experimental group and the control group is consistent) and the experiment is isolated, otherwise it will affect the experimental results and cause wrong conclusions.

[0014] In order to measure and optimize the performance of intelligent models, scientific and systematic evaluation can be performed on them. Traditional evaluation methods have the disadvantages of data bias, index limitation, low efficiency and high cost. Embodiments of the present specification propose an agent evaluation mechanism based on a sandbox driven by a dynamic game algorithm. By constructing a highly simulated virtual environment, the trading environment of a certain duration can be restored in a short time (minute-hour level) for intelligent models (such as pricing models), hosting strategies and recommendation algorithms, and the algorithm effect can be comprehensively tested and evaluated. This sandbox environment can simulate various conventional or extreme trading conditions, user behaviors and competitive situations, so as to quickly verify the effectiveness and risk of the agent in a safe and controllable environment, and drive the iterative optimization of the algorithm model. This method not only improves the accuracy and robustness of the model, but also significantly reduces the risk and cost of testing in the real environment through early estimation and verification. Specific processing can be referred to the specific content in the following embodiments.

[0015] The evaluation method of the business agent provided by one or more embodiments of the present specification can be applied to the implementation environment of evaluating business agents, referring to Figure 1 The implementation environment at least includes: The client 100 and the server 200. In addition, the server 200 can include a plurality of different types of evaluation systems, which can include a plurality of different algorithms, strategies, rules, models, and even agents, wherein: The client 100 can run on a terminal device, which can be a mobile phone, a personal computer, a tablet computer, an e-book reader, a wearable device, a device for information interaction based on AR (Augmented Reality) and VR (Virtual Reality), and a laptop computer, etc. The terminal device can install the client 100, which can be an application program, a browser or a subprogram loaded in the application program, etc.

[0016] The server 200 can run on a server, which can be one or more servers, a server cluster composed of several servers, or a cloud server of a cloud computing platform, etc. The server can install the server 200, which can be an application program or a subprogram loaded in the application program, etc. A plurality of different algorithms, strategies, rules, models, and even agents can be integrated in the server 200, or the server 200 can call any one or more of the plurality of different algorithms, strategies, rules, models, and even agents to perform corresponding operations.

[0017] In addition, the database 300 can be arranged in the server running the service end 200 or outside the server running the service end 200, and the database 300 can store information related to the specified business, such as commodity information, price range, and historical sales information in the determined sales business.

[0018] In this implementation environment, the client 100 can generate an evaluation request for the target business agent, and send the evaluation request to the service end 200, wherein the target business agent is used for performing a preset business prediction process on a target business that needs to be deployed by the target business agent. The service end 200 can copy business traffic data from the target business that needs to be deployed by the target business agent, and based on the business traffic data, perform a business prediction process through the target business agent to obtain a corresponding business prediction result. Then, the target business agent can be taken as a leader, and a preset benchmark prediction model can be taken as a follower. Based on the historical business trend information in the target business and the business prediction result, the target business corresponding competitive situation simulation information is determined through a preset dynamic game model. Finally, based on the target business corresponding competitive situation simulation information and the business prediction result, the target business agent needed to be deployed to the target business is evaluated through a preset business estimation model to obtain a corresponding evaluation result. By constructing a highly simulated virtual environment, the algorithm effect can be comprehensively tested and evaluated in a certain length of transaction environment (minute-hour level) in a short time (minute-hour level). The sandbox environment can simulate various conventional or extreme trading conditions, user behaviors and competitive situations, so as to quickly verify the effectiveness and risk of the intelligent agent in a safe and controllable environment, and drive the iterative optimization of the algorithm model. This method not only improves the accuracy and robustness of the model, but also significantly reduces the risk and cost of testing in the real environment through early estimation and verification.

[0019] As shown in Figure 2 The business agent evaluation method provided by the embodiments of the present specification can be executed by a terminal device or a server, etc. The terminal device can be a mobile terminal device such as a mobile phone or a tablet computer, or a computer device such as a notebook computer or a desktop computer, or an IoT device (such as a smart watch or a vehicle-mounted device, etc.). The server can be a single server or a server cluster composed of multiple servers. The server can be a background server of a financial business or an online shopping business, or a background server of an application program, etc. In this embodiment, the execution subject is taken as a server for detailed description. For the case where the execution subject is a terminal device, the case of the server can be referred to, and details are not repeated here. The method can specifically include the following steps: In step S202, a request for evaluating the target business agent is received, and the target business agent is used for preset business prediction processing of a target business that needs to be deployed by the target business agent.

[0020] The target business agent can include various algorithms, models, and large models, etc. Different functions can be achieved by different algorithms, models, and large models, etc. in the target business agent, and the preset business prediction processing of the target business can be achieved by the cooperation of the algorithms, models, and large models. For example, the target business agent can be an agent for predicting or determining the selling price of a certain commodity, or the target business agent can be an agent for information recommendation, etc. The specific setting can be made according to the actual situation. The target business can be any business, for example, the target business can be a commodity pricing, coupon pricing business, etc. in a merchant intelligent operation scene, or an information recommendation business, etc. The specific setting can be made according to the actual situation.

[0021] In implementation, when a certain business agent (i.e. the target business agent) needs to be evaluated, the service program of the agent evaluation in the server can be started. The server can display the home page of the service program, and the home page can be provided with information of various types of agents, functions or uses of agents, etc. The tester can determine the type to which the target business agent belongs, and then trigger the agent evaluation process. At this time, the server can obtain the target business agent to be evaluated. In addition, the tester can also add other evaluation requirements (such as evaluation time, evaluation environment, etc.). At this time, the server can generate an evaluation request for the target business agent by combining the above-mentioned other evaluation requirements, the target business agent to be tested, etc. The server can receive the evaluation request for the target business agent. Alternatively, the tester can use the terminal device to trigger the server to perform the agent evaluation processing. That is, when a certain business agent (i.e. the target business agent) needs to be evaluated, the tester can start the application program of the agent evaluation in the terminal device. The terminal device can display the home page of the application program, and the home page can be provided with information of various types of agents, functions or uses of agents, etc. The tester can determine the type to which the target business agent belongs, and then trigger the agent evaluation process. At this time, the server can obtain the target business agent to be evaluated. In addition, the tester can also add other evaluation requirements (such as evaluation time, evaluation environment, etc.). The terminal device can also generate an evaluation request for the target business agent by combining the above-mentioned other evaluation requirements, the type of the agent to be tested, etc. and send the evaluation request to the server. The server can receive the evaluation request for the target business agent.

[0022] In step S204, the business traffic data is copied from the target business required to be deployed by the target business agent, and based on the business traffic data, the business prediction processing is performed by the target business agent to obtain the corresponding business prediction result.

[0023] The business traffic data can be traffic data generated for the target business, and can include various different requests, user search information, and returned information for the target business, etc., which can be set according to actual conditions.

[0024] In implementation, in order to simulate the near-real scene of the field traffic environment, for example, the whole logic of user request→data recall→target business agent (such as pricing model, etc., according to the click conversion rate estimated by RTP)→business prediction can be included, a sandbox environment simulating the running of a real system can be constructed, which allows software, programs or codes to run in a closed setting without affecting external systems or environments. A simulation engine can be set in the sandbox environment, which is a framework basic capability for injecting basic traffic into the simulation sandbox. The whole logic of user request→data recall→target business agent (such as pricing model, etc., according to the click conversion rate estimated by RTP)→business prediction is simulated, and its core module can include traffic data cleaning and construction, task scheduling, traffic data accelerated playback, feature backflow, etc. The balance between the fidelity of the environment and the simulation cost must be considered here: ideally, 100% reuse of production links can completely reflect the real scene, but the simulation traffic data will pollute the production environment (such as database and cache writing, data feature backflow, etc.) and break the production safety bottom line; 0% reuse to rebuild a completely isolated simulation environment not only has a huge cost, but also causes the system layer version configuration to be out of sync, the data layer features to be distorted, and other problems that cause the simulation accuracy to drop sharply, losing the simulation significance. Therefore, the read and write in the business link can be processed in stages, and the lossless read traffic data can reuse the production system, and the lossy write traffic data can be routed to the corresponding isolated link. Specifically, real-time or specified time period business traffic data can be obtained from the target business required to be deployed by the target business agent, and the above business traffic data can be copied. The copied business traffic data can be put into the above sandbox environment. In the sandbox environment, the business traffic data can be input into the target business agent, and the target business is processed by the algorithm, model or large model in the target business agent to output the corresponding business prediction result.

[0025] In step S206, the target business agent is taken as the leader, and the preset benchmark prediction model is taken as the follower. Based on the historical business change trend information in the target business and the above business prediction result, the target business corresponding competition situation simulation information is determined by the preset dynamic game model.

[0026] Historical business trend information can be the business trend of the target business within a specified time period. This trend can vary depending on the target business. For example, if the target business is product pricing in a smart business scenario, the trend could be the trend of product pricing or product selling price, etc., and can be set according to the specific situation. For the dynamic game model, assume there are two institutions whose decision-making is output. One institution is dominant and acts first to choose its output, while the other is subordinate and makes its choice after the dominant institution has chosen its output. Therefore, this is a dynamic game problem, and other aspects of the game structure can be further assumed, such as the strategy space, payoff function, and information structure. In practical applications, dynamic game models can include the Stackelberg dynamic game model, etc. The baseline prediction model can be a model that corresponds to the target business agent and has the same functions as the target business agent. In practical applications, the baseline prediction model can also be presented in the form of an agent, namely the baseline prediction agent. In practical applications, the baseline prediction model can be constructed in a variety of different ways. For example, the baseline prediction model can be constructed through expert experience, or it can be a business model with good business performance obtained through other means and used as the baseline prediction model, etc. The specific settings can be set according to the actual situation.

[0027] In implementation, such as Figure 3 As shown, to evaluate the performance of the target business agent, a benchmark prediction model can be introduced. In a sandbox environment simulating traffic data, both the benchmark prediction model and the target business agent react and make decisions based on changes in the business environment and competitors' business in order to maximize the specified business objective. This competitive game process can be abstracted into a dynamic game model. Specifically, the target business agent is the leader (i.e., in a dominant position), and the preset benchmark prediction model is the follower (i.e., in a subordinate position). Historical business change trend information and the aforementioned business prediction results can be input into the preset dynamic game model. The leader in the dynamic game model makes business choices by taking the first action (e.g., ...). Figure 3 If random events occur (such as missing or abnormal simulation data, which can increase the difficulty of evaluation), the leader can take the lead in making business choices (such as adjusting prices), which may lead to business changes (such as price changes). Followers then make their choices after the leader has made its business choices (such as...). Figure 3 The follower (i.e., one side of the baseline prediction model) responds to the dominant player's business choices (such as adjusting prices) according to the set competitive strategy, and makes corresponding business adjustments (such as adjusting prices) to determine the competitive situation simulation information corresponding to the target business.

[0028] In step S208, based on the competitive situation simulation information corresponding to the target business and the above business prediction results, the target business agent to be deployed in the target business is evaluated through a preset business prediction model to obtain the corresponding evaluation results.

[0029] Among them, the business prediction model can be a model used to predict business results. The business prediction model can be constructed using a specified algorithm or network. For example, the business prediction model can be constructed using a convolutional neural network or a recurrent neural network, etc. The specific configuration can be set according to the actual situation.

[0030] In implementation, the aforementioned competitive processing based on the dynamic game model can affect the business performance of the target business. To address the issue of predicting business performance gains in a sandbox environment, a business prediction model based on competitive dynamics and business performance gains is proposed and used as a real-time feature for feedback. Specifically, for example... Figure 3 As shown, the competitive landscape simulation information corresponding to the target business and the aforementioned business prediction results can be input into the business prediction model. Through the business prediction model, combined with the competitive landscape simulation information corresponding to the target business and after probability correction, the target business agent to be deployed in the target business is evaluated, yielding the final evaluation result. Statistical analysis can also be performed on the obtained evaluation results to determine various information, such as order revenue information, abnormal bidding information, and the existence of certain risks. Based on the above statistical analysis, it can be determined whether the target business agent passes the evaluation.

[0031] The embodiment of the present specification provides a service intelligent agent evaluation method, by receiving an evaluation request for a target service intelligent agent, copying service traffic data from a target service required to be deployed by the target service intelligent agent, and based on the service traffic data, performing service prediction processing through the target service intelligent agent to obtain a corresponding service prediction result, then taking the target service intelligent agent as a leader and a preset benchmark prediction model as a follower, based on historical service trend information and the service prediction result in the target service, determining the corresponding competitive situation simulation information of the target service through a preset dynamic game model, finally, based on the corresponding competitive situation simulation information and the service prediction result of the target service, evaluating the target service intelligent agent required to be deployed in the target service through a preset service estimation model to obtain a corresponding evaluation result, in this way, by constructing a highly simulated virtual environment, the algorithm effect can be comprehensively tested and evaluated in a certain length of transaction environment (minute-hour level) of the intelligent model (such as a pricing model), hosting strategy and recommendation algorithm, and the sandbox environment can simulate various conventional or extreme transaction conditions, user behaviors and competitive situations, so as to quickly verify the effectiveness and risk of the intelligent agent in a safe and controllable environment, and drive the iteration and optimization of the algorithm model, which can not only improve the accuracy and robustness of the model, but also significantly reduce the risk and cost of testing in the real environment through early estimation and verification.

[0032] In actual application, the specific processing manner of the step S204 of performing service prediction processing through the target service intelligent agent based on the service traffic data to obtain the corresponding service prediction result can be various, and the following provides an optional processing manner, which can specifically include the processing of the steps S2042-S2048, based on which, on the basis of the above Figure 2 , the steps included in the method can be as shown in Figure 4 .

[0033] In the step S2042, a shadow link for evaluating the target service intelligent agent is constructed, and the shadow link is isolated from the processing link of the target service.

[0034] The shadow link can be a link whose production environment is different from the formal traffic data (i.e., the link of the real service processing of generating various request traffic data, that is, the processing link of the target service), in order to ensure the fidelity of the simulation data, the production link playback is required, and the system, storage and middleware level are required to be isolated from the traffic data for the purpose of not polluting the production environment, for example, the identified simulation write traffic data is required to be routed to the shadow database instead of being written into the actual database of the target service.

[0035] In implementation, in order to solve the balance between the fidelity of the environment and the simulation cost, a shadow link is proposed, through which a "read multiplexing + write shadow" scheme can be implemented, that is, the read operation and the write operation in the business processing link are processed in a hierarchical manner, in which the "read multiplexing" can realize the read operation multiplexing production system of the traffic data without loss, the write operation of the traffic data with loss, and routing it to the shadow link. Based on this, a shadow link for evaluating the target business agent can be constructed, which is isolated from the processing link actually used by the target business. In this way, a sandbox environment is constructed, which provides a basic environment for agent evaluation. The fidelity of the sandbox environment largely determines the accuracy of the evaluation result. The simulation engine is the core module of the sandbox environment, and is a basic capability of the entire traffic data construction, task scheduling, traffic data playback, shadow link, business effect feature backflow, etc.

[0036] Under the shadow link, the following steps S2044-S2048 are executed: In step S2044, the business traffic data is marked to identify the business traffic data as traffic data for evaluating the target business agent, and the label information corresponding to the business traffic data is obtained.

[0037] In implementation, for the business traffic data that needs to be marked, it can be obtained in the manner as shown in Figure 5 , that is, through sample traffic construction and traffic playback, etc. Finally, the business traffic data that needs to be marked is determined, wherein the process of sample traffic construction can include sample source and sample processing. The sample source can include various data and processing mechanisms, such as various different request logs, request construction mechanisms, request generation mechanisms, search and push logs (i.e. logs recording user search information and corresponding push information, etc.), and sample processing can include log collection (i.e. collecting logs from the sample source) and log processing (i.e. processing the collected logs) to determine the business traffic data that needs to be marked. Figure 5In the manner of SLS collection in the foregoing embodiment, a certain amount of logs are collected, and then the collected logs can be aggregated and filtered, and the filtered logs are stored in the database hbase after being sharded. The traffic playback processing can be driven by the corresponding execution engine, the execution engine can schedule the current task queue, and then the sample record can be performed, at this time a certain amount of logs can be pulled from the database hbase, and the pulled logs can be loaded, after the loading is completed, the logs can be analyzed, and the corresponding request in the logs can be reconstructed, when the business traffic data needs to be copied, the request can be obtained by calling the route, the obtained request can be used as the business traffic data, in order to distinguish it from the normal process data in the target business, the business traffic data can be marked in the middleware layer to identify the business traffic data as the traffic data for evaluating the target business agent, and the label information (such as the pressure test label, the simulation label, and the UID of the pressure test) corresponding to the business traffic data is obtained, so as to realize traffic data isolation and identification, which realizes read reuse of the production link and isolates the pollution factors through the shadow link, and solves the balance problem between fidelity and simulation cost.

[0038] In step S2046, the business traffic data with label information is preprocessed to obtain preprocessed business traffic data, and the preprocessing includes filtering and / or data modification processing.

[0039] In implementation, the information in the business traffic data with label information may not necessarily completely match the current agent evaluation, at this time, the business traffic data with label information can be filtered to filter out invalid or meaningless business traffic data in the business traffic data with label information, in addition, the business traffic data with label information can be subjected to data modification processing, for example, the time recorded in the business traffic data with label information can be modified to the current time, or a certain specified parameter in the business traffic data with label information can be modified to a set parameter value, etc., which can be set according to actual conditions. In actual application, in order to avoid pollution of production link data, for lossy traffic data, once the above label information is identified, special logic processing can be performed, for example, the lossy components in the target business are removed (for example, the simulation traffic data is not deducted from the budget), in addition, the special components are called by the online, for example, the query of user features can directly query the database, the traffic data of the write operation is routed to the shadow database instead of being written into the database corresponding to the business processing of the target business, in addition, the business logic component can be modified, for example, the normal logic is that a certain commodity is not recalled at the selling time, which needs to be modified to recall otherwise it will be distorted, which can be set according to actual conditions.

[0040] In step S2048, based on the pre-processed service traffic data, the target service agent performs service prediction processing to obtain a corresponding service prediction result, and stores the service prediction result into the shadow database corresponding to the shadow link.

[0041] The specific processing mode of step S2048 can refer to the foregoing related content, which will not be repeated here.

[0042] Through the construction of the shadow link and various processing procedures under the shadow link, the consistency rate of the request playback engine and the request online engine can be observed, and the request playback engine and the request online engine have a consistency rate of 95% in the final set return, so as to ensure the accuracy of the evaluation effect of the target service agent.

[0043] In actual application, other service processing can be performed on the target service, and the target service corresponding competitive situation simulation information and the like can be combined to evaluate the target service agent. Specifically, the following step S210 can be included, and based on the above Figure 4 , the method specifically includes the steps as shown in Figure 6 .

[0044] In step S210, under the shadow link, based on the service prediction result, the target service corresponding service processing is performed to obtain corresponding service processing data; and the service processing data is stored into the shadow database.

[0045] In implementation, as shown in Figure 5 , the target service can also be processed based on the obtained service prediction result. For example, based on the obtained service prediction result, the service prediction result can be sorted with other related data, and then the sorted data can be adjusted (such as fine-tuning the sorting of the specified data, etc.). Through the above processing, the corresponding service processing data can be obtained, and the service processing data can be stored into the shadow database as simulation data.

[0046] Correspondingly, the specific processing mode of step S208 can be various, and the following provides an optional processing mode, which can include the following steps S2082 and S2084.

[0047] In step S2082, the service prediction result and the service processing data are obtained from the shadow database, and the feature extraction is performed on the service prediction result and the service processing data to obtain corresponding data features.

[0048] In implementation, as shown in Figure 5As shown, the business prediction result and the business processing data can be obtained from the shadow database, and the business prediction result and the business processing data can be subjected to feature extraction through feature engineering to obtain corresponding data features.

[0049] In step S2084, based on the competition situation simulation information corresponding to the target business and the obtained data features, the target business agent required to be deployed into the target business is evaluated through a preset business estimation model to obtain a corresponding evaluation result.

[0050] In implementation, as shown in Figure 5 The competition situation simulation information corresponding to the target business and the obtained data features can be input into the business estimation model, and the target business agent required to be deployed into the target business is evaluated through the business estimation model to obtain a corresponding evaluation result.

[0051] In actual application, the specific processing mode of the above step S206 can be various, and the following provides an optional processing mode, which can specifically include the processing of steps S20602-S20610, based on which, on the basis of the above Figure 6 The steps included in the method can be as shown in Figure 7

[0052] In step S20602, the target business agent is taken as a leader, and a preset benchmark prediction model is taken as a follower to obtain the decision range of the leader and the decision range of the follower.

[0053] In implementation, in order to simulate the behavior of maximizing the effect of the target business in the target business field, i.e., the expert operator observes the business change, the competitor adjusts the business change, and the target business agent adjusts the output business prediction result according to the historical data and the specified data features, a dynamic game model is used to model this process, and the problem abstraction and scheme principle are described as follows: the target business agent is taken as a leader, and a preset benchmark prediction model is taken as a follower; based on the observed business change and the relevant business of the competitor, both sides decide the optimal business prediction result to maximize their GMV. The decision range of the leader and the decision range of the follower can be obtained, and the decision range can be the change range of a certain business index, which can be different according to the target business, for example, the target business is the commodity pricing business in the intelligent business scene of the merchant, and the decision range can be the pricing range. As shown in Figure 8 The decision range of the leader can be [0, P1], and the decision range of the follower can be [0, P2].

[0054] ​In step S20604, the decision information of the leader is determined based on the decision range of the leader and the business prediction result of the leader, and the business prediction result corresponding to the leader is generated by the target business agent based on the decision information of the leader.

[0055] In implementation, in the dynamic game model, the decision sequence of the target business can make the corresponding business strategy (such as price strategy or production strategy, etc.) for the leader, and the follower can observe the business decision of the leader and then make corresponding adjustment to its related business. Based on this, the decision information of the leader can be generated based on the decision range of the leader and the business prediction result of the leader. For example, the obtained decision information can be to lower the value of a specified business parameter in the target business. For example, if the target business is a commodity pricing business in the intelligent business scenario of a merchant, the selling price of the target business can be lowered, etc. The decision information of the leader can be input into the target business agent, and the business prediction result corresponding to the leader is generated by the target business agent p1.

[0056] In step S20606, the current competition situation information is obtained by the benchmark prediction model based on the business prediction result corresponding to the decision information and the historical business trend information in the target business, and the follower reaction function is constructed based on the current competition situation information.

[0057] The historical business trend information in the target business can include historical change data of a specified parameter in the target business, such as historical change data of the selling price of a certain commodity in the target business, etc. The follower reaction function can be a function of the follower adjusting according to the business decision of the leader. The follower reaction function can be constructed in various ways, such as a linear function or a specified nonlinear function, etc. The follower reaction function can be formulated according to the following strategies: profit maximization or sacrificing profit for market share, etc. The specific strategy can be set according to the actual situation.

[0058] In implementation, as shown in Figure 8 The business prediction result corresponding to the decision information and the historical business trend information in the target business can be input into the benchmark prediction model, the current competition situation information is predicted by the benchmark prediction model, and then the follower reaction function can be constructed based on the current competition situation information.

[0059] In step S20608, the business prediction result corresponding to the follower is generated by the follower reaction function based on the business prediction result corresponding to the decision information.

[0060] In implementation, as shown in Figure 8As shown, followers can analyze the decision information of the leader and the corresponding business forecast results, and then generate the business forecast result p2 corresponding to the follower through the follower reaction function.

[0061] In step S20610, based on the business forecast results corresponding to the leader and the business forecast results corresponding to the follower, the competitive situation simulation information corresponding to the target business is determined.

[0062] In implementation, such as Figure 8 As shown, in the dynamic game model, the dominant player often has a first-mover advantage, while the follower can optimize business performance through follower optimization. The final business performance can be measured by methods such as the number of user orders and / or profit distribution. The specific method can be set according to the actual situation, and the embodiments in this specification do not limit this.

[0063] To make the dynamic game process described above clearer, a specific example is provided below for detailed explanation: Condition 1: The marginal cost prices of the leader and the follower are set at 1.5 and 1.8, respectively; Condition 2: Price competitiveness G(p1,p2) = 0.6 * p1 - 0.8 * p2 + 2. For simplicity, a linear function is used here: for every unit decrease in price, competitiveness increases by 0.8 units; for every unit increase in competitor price, competitiveness increases by 0.6 units; the basic competitiveness level is 2 units.

[0064] Condition 3: Sales volume and price competitiveness are directly proportional Q(g) = 50*g + 100. For simplicity, a linear sales response is used here.

[0065] The market competition model can be abstracted into a Stackelberg game problem, and when both parties aim to maximize GMV, a better price decision can be made based on the above conditions.

[0066] The problem-solving process of dynamic game theory models: 1. Based on the above conditions, the profit function U = price * sales volume is a simple quadratic function:

[0067]

[0068] 2. Find the one that makes U Maximizing p2 and finding the derivative yields the price at which U2 is maximized, which is also the follower's reaction function:

[0069] 3. The leader first decides the follower's reaction and maximizes his own profit curve, which is substituted into U 1, and the derivative is taken to get .

[0070] The follower then decides after observing and decides

[0071] 4. The final decision prices and profits of both sides are shown in Table 1 as follows: Table 1

[0072] The above describes how the dynamic game model drives the decision-making of both sides to set prices through a simplified version. The modeling in the actual process is usually more complex, such as optimization coping strategies: target optimization function:

[0073] wherein, represents the price; represents a price sensitivity constant, which can be solved by the effect of the last round; represents the total sales of similar goods (current goods + competitors) (which can use the last round of data); represents the current goods; represents the competitors; represents the weight of the competitors, which can be estimated by the sales of the competitors i in the last round / all competitor sales. Through the above formula, the (i.e. the price of the current goods) when the GMV is maximum can be solved.

[0074] The last round of effect data can be used as input data, substituted into the following formula

[0075] to solve the price sensitivity constant .

[0076] In actual application, the business estimation model can be obtained by model training in the following manner, and specific processing can be referred to steps A2 to A6.

[0077] In step A2, sample data for training the business estimation model is obtained, and the sample data includes historical business data related to the target business and business prediction result samples obtained by business prediction processing of the target business agent.

[0078] In implementation, business competition can affect exposure, user clicks, and order intention of a business, and further affect business effect. In order to solve the estimation problem of business effect in a sandbox environment (including changes in exposure rate, click rate, and order after the business changes, and business profit GMV income), a modeling method of a business estimation model based on business changes and business effect gain is proposed, and is fed back to the target business agent as real-time feature. For this purpose, the historical business data related to the target business can be obtained, and the business prediction result sample obtained by the target business agent for business prediction processing can be obtained, and the above obtained data can be used as sample data for training the business estimation model.

[0079] In step A4, according to different function information of the business estimation model, a model architecture matched with each function information is obtained from a preset business modeling database, and a business estimation model is constructed based on the obtained model architecture.

[0080] Among them, the business modeling database can include a plurality of different model architectures, for example, the business modeling database can include a NN network (i.e. neural network), a time series prediction model, a user willingness prediction architecture, and a target operation optimization architecture, etc. The specific settings can be made according to actual conditions.

[0081] In implementation, each function of the business estimation model can be analyzed to determine different function information of the business estimation model. For each function information, a model architecture matched with the function information can be obtained from a preset business modeling database, so as to obtain a model architecture matched with each function information. The obtained model architecture can be spliced and combined according to different function information of the business estimation model to obtain the overall architecture of the business estimation model.

[0082] In step A6, based on the historical business data and the business prediction result sample, the business estimation model is trained to obtain a trained business estimation model.

[0083] In implementation, the model training of the business estimation model can be completed by offline training, online training, or a combination of offline training and online training. In addition, after obtaining the trained business estimation model, the business estimation model can be updated regularly or irregularly, and the model update process can also be completed by incremental update, etc. The specific settings can be made according to actual conditions.

[0084] In actual application, the business estimation model can also be trained in combination with the business processing data sample. The business processing data sample is obtained in the following manner, specifically including the following contents: obtaining the business processing data sample obtained by performing the target business based on the business prediction result sample under the shadow link, the business processing data sample including the simulation information of the processing strategy of the target business, the competition simulation information corresponding to the target business, the data fluctuation situation information of the first preset number of business traffic data samples copied from the target business, and the strategy information of the competitor of the preset commodity corresponding to the target business.

[0085] Correspondingly, the processing of step A6 can include: training the business estimation model based on the historical business data, the business prediction result sample and the business processing data sample to obtain the trained business estimation model.

[0086] In actual application, the target business can be a commodity pricing business or a coupon pricing business in a merchant intelligent operation scenario, and the target business agent is a business agent for determining the commodity selling price; the business traffic data includes one or more of various request data for a preset commodity, user input search information related to the preset commodity, and push information returned for the search information; and the business estimation model is a model for performing revenue estimation processing based on the commodity selling price determined by the target business agent.

[0087] In actual application, based on the specific scenario of the target business (i.e., the target business can be a commodity pricing business or a coupon pricing business in a merchant intelligent operation scenario), the historical business data in the process of training the business estimation model can include one or more of the historical field traffic data of the target business, the historical selling price ranking information corresponding to the target business, the characteristics of the preset commodity, the characteristics of the user, the traffic characteristics of the same model or same family commodity, and the sales characteristics of the same model or same family commodity; and the competition simulation information corresponding to the target business includes the bidding simulation information corresponding to the target business.

[0088] Based on the above content, the training of the business estimation model can be referred to as shown in Figure 9 The business estimation model measures the revenue brought by the bidding result through the business estimation model. Bidding will affect the exposure ranking of the commodity, the user clicks, the willingness to make a single, and thus affect the business effect. In order to solve the problem of revenue estimation in the sandbox environment (including the changes of exposure rate, click rate and single after the change of commodity price, and the business profit GMV revenue, etc.), the present scheme proposes a modeling method of the business estimation model based on the change of selling price and the gain of business effect, and feeds back to the target business agent as real-time feature. The training of the business estimation model combines the historical features corresponding to the historical business data and the features of the competition simulation information corresponding to the target business as the input data of the model training: historical features (orFigure 9 The historical fact features in the target business can include product (such as category, same product, etc.) features, merchant features, and business effect features before and after price adjustment (such as exposure rate, click rate, conversion rate, etc.). The features of the competitive simulation information corresponding to the target business can include simulation field position ranking changes, exposure fluctuations, and traffic data fluctuations.

[0089] As shown in Figure 9 , simulation feature factors can also be included. The simulation feature factors are pre-event feature factors and cannot be obtained from online data. In addition, the model training and correction part and the business modeling and optimization part are also included. The model training and correction part can provide offline training, online training, and other training methods, and can also provide probability correction, normalization processing, model incremental updating, etc. The business modeling and optimization part can provide business modeling databases including user willingness prediction, NN network, time series prediction model, multi-objective operational optimization, etc. The business modeling and optimization part can also optimize the business estimation model through multi-objective modeling, target control, etc.

[0090] In actual application, the copied business traffic data includes multiple, at this time, the comprehensive evaluation of the business effect of the target business agent can also be realized through N rounds of circulation. Based on this, the specific processing mode of the above steps S206 and S208 can be various, and the following will provide an optional processing mode, which can include the following steps S212-S216. Based on the above Figure 7 , the method specifically includes the steps as shown in Figure 10 .

[0091] In step S212, for any two adjacent business traffic data in the multiple business traffic data, the business traffic data arranged in front is taken as the leader with the target business agent as the follower, and a preset baseline prediction model is taken as the follower. Based on the historical business trend information and the business prediction result in the target business, the competitive situation simulation information corresponding to the target business is determined through a preset dynamic game model. Based on the competitive situation simulation information corresponding to the target business and the business prediction result, the target business agent needed to be deployed in the target business is evaluated through a preset business estimation model, and an evaluation result corresponding to the business traffic data arranged in front is obtained.

[0092] In implementation, the target business can be taken as an example of a commodity pricing business or a coupon pricing business in a merchant intelligent operation scene, as shown in Figure 11As shown, when starting the evaluation, the adjustment number N of the selling price, the price competition mode (i.e. the bidding mode), and the business target (such as the maximum revenue) can be taken as the input data, and then the analysis of the existing competition goods can be performed. The copied business flow data is input into the target business agent and the benchmark prediction model. Specifically, for the business flow data arranged in the front, the target business agent is taken as the leader, and the preset benchmark prediction model is taken as the follower. Based on the historical business change trend information and the business prediction result in the target business, the competition situation simulation information corresponding to the target business is determined through the preset dynamic game model. Based on the competition situation simulation information corresponding to the target business and the business prediction result, the target business agent required to be deployed to the target business is evaluated through the preset business estimation model, and the evaluation result corresponding to the business flow data arranged in the front is obtained. The specific processing can be referred to the foregoing related content, and will not be described here.

[0093] In step S214, for the business flow data arranged in the back, the target business agent is taken as the leader, and the preset benchmark prediction model is taken as the follower. Based on the historical business change trend information, the business prediction result, and the evaluation result corresponding to the business flow data arranged in the front, the competition situation simulation information corresponding to the target business is determined through the preset dynamic game model. Based on the competition situation simulation information corresponding to the target business and the business prediction result, the target business agent required to be deployed to the target business is evaluated through the preset business estimation model, and the evaluation result corresponding to the business flow data arranged in the back is obtained.

[0094] In implementation, referring to Figure 11 For the business flow data arranged in the back, the above processing is repeated, that is, the target business agent is taken as the leader, and the preset benchmark prediction model is taken as the follower. Based on the historical business change trend information, the business prediction result, and the evaluation result corresponding to the business flow data arranged in the front, the competition situation simulation information corresponding to the target business is determined through the preset dynamic game model. Based on the competition situation simulation information corresponding to the target business and the business prediction result, the target business agent required to be deployed to the target business is evaluated through the preset business estimation model, and the evaluation result corresponding to the business flow data arranged in the back is obtained. The specific processing can be referred to the foregoing related content, and will not be described here.

[0095] In step S216, the evaluation result corresponding to each business flow data in the plurality of business flow data is obtained based on the above processing mode, and the evaluation result of the target business agent required to be deployed to the target business is determined based on the evaluation result corresponding to each business flow data in the plurality of business flow data.

[0096] In implementation, referring to Figure 11, through the above-mentioned loop iteration, the business effect calculated after N rounds of loop combination can be obtained, and then the final incremental benefit of the business effect can be obtained, in addition, the selling price of the key process and the analysis data of the summary result can be obtained, and then the robustness, rationality, data distribution and other indicators of the model can be determined, and subsequently the evaluation result of the target business agent to be deployed to the target business can be determined based on the final incremental benefit of the business effect, the robustness, rationality, data distribution and other indicators of the model.

[0097] The embodiment of the present specification provides a business agent evaluation method, by receiving an evaluation request for a target business agent, copying business traffic data from a target business to which the target business agent needs to be deployed, and based on the business traffic data, performing business prediction processing through the target business agent to obtain corresponding business prediction results, then, taking the target business agent as the leader and taking the preset benchmark prediction model as the follower, based on the historical business change trend information and the business prediction results in the target business, determining the corresponding competitive situation simulation information of the target business through the preset dynamic game model, finally, based on the corresponding competitive situation simulation information and the business prediction results of the target business, evaluating the target business agent to be deployed to the target business through the preset business estimation model, and obtaining the corresponding evaluation result, in this way, by constructing a highly simulated virtual environment, the algorithm effect can be tested and evaluated comprehensively in a certain length of transaction environment in a short time (minute-hour level), the sandbox environment can simulate various conventional or extreme transaction conditions, user behavior and competitive situation, so as to quickly verify the effectiveness and risk of the intelligent agent in a safe and controllable environment, and drive the iterative optimization of the algorithm model, this way not only can improve the accuracy and robustness of the model, but also can significantly reduce the risk and cost of testing in the real environment through early estimation and verification.

[0098] The above is the business agent evaluation method provided by the embodiment of the present specification, based on the same idea, the embodiment of the present specification also provides a business agent evaluation device, as shown in Figure 12 .

[0099] The business agent evaluation device comprises an evaluation request module 1201, a business prediction module 1202, a business prediction module 1203 and an agent evaluation module 1204, wherein: The evaluation request module 1201 receives an evaluation request for a target business agent, and the target business agent is used for performing a preset business prediction processing on a target business to which the target business agent needs to be deployed; The service prediction module 1202 copies service traffic data from a target service to which the target service agent needs to be deployed, and performs service prediction processing on the target service agent based on the service traffic data, to obtain a corresponding service prediction result. The competition simulation module 1203 takes the target service agent as a leader and a preset benchmark prediction model as a follower, determines competition situation simulation information corresponding to the target service based on historical service change trend information in the target service and the service prediction result, through a preset dynamic game model. The agent evaluation module 1204 evaluates the target service agent needed to be deployed to the target service based on the competition situation simulation information corresponding to the target service and the service prediction result, through a preset service estimation model, to obtain a corresponding evaluation result.

[0100] In the embodiments of the present specification, the service prediction module 1202 comprises: A shadow link construction unit constructs a shadow link used for evaluating the target service agent, and the shadow link is isolated from a processing link of the target service. The following processing is performed under the shadow link: A marking unit performs marking processing on the service traffic data, to identify the service traffic data as traffic data used for evaluating the target service agent, to obtain label information corresponding to the service traffic data. A preprocessing unit pre-processes the service traffic data with the label information, to obtain pre-processed service traffic data, and the preprocessing comprises filtering processing and / or data modification processing. A service prediction unit performs service prediction processing on the pre-processed service traffic data through the target service agent, to obtain a corresponding service prediction result, and stores the service prediction result in a shadow database corresponding to the shadow link.

[0101] In the embodiments of the present specification, the apparatus further comprises: A service processing module performs service processing corresponding to the target service based on the service prediction result under the shadow link, to obtain corresponding service processing data, and stores the service processing data in the shadow database. The agent evaluation module 1204 comprises: A feature extraction unit obtains the service prediction result and the service processing data from the shadow database, and extracts features of the service prediction result and the service processing data, to obtain corresponding data features. The agent evaluation unit evaluates the target service agent to be deployed in the target service based on the competition situation simulation information corresponding to the target service and the obtained data characteristics, through a preset service estimation model, to obtain a corresponding evaluation result.

[0102] In the embodiments of the present specification, the competition simulation module 1203 comprises: The information acquisition unit acquires the decision range of the leader and the decision range of the follower, with the target service agent as the leader and a preset benchmark prediction model as the follower. The decision unit determines the decision information of the leader based on the decision range of the leader and the service prediction result, and generates the service prediction result corresponding to the leader through the target service agent based on the decision information of the leader. The competition situation determination unit acquires current competition situation information through the benchmark prediction model based on the service prediction result corresponding to the decision information and historical service trend information in the target service, and constructs a follower reaction function based on the current competition situation information. The follower prediction unit generates the service prediction result corresponding to the follower through the follower reaction function based on the service prediction result corresponding to the decision information. The competition simulation unit determines the competition situation simulation information corresponding to the target service based on the service prediction result corresponding to the leader and the service prediction result corresponding to the follower.

[0103] In the embodiments of the present specification, the device further comprises: The first sample acquisition module acquires sample data for training the service estimation model, wherein the sample data comprises historical service data related to the target service and a service prediction result sample obtained through service prediction processing of the target service agent. The model architecture determination module acquires a model architecture matched with each function information from a preset service modeling database according to different function information possessed by the service estimation model, and constructs the service estimation model based on the acquired model architecture. The training module performs model training on the service estimation model based on the historical service data and the service prediction result sample, to obtain a trained service estimation model.

[0104] In the embodiments of the present specification, the device further comprises: a second sample obtaining module, which obtains a service processing data sample obtained by performing service processing corresponding to the target service based on the service prediction result sample under the shadow link, the service processing data sample including simulation information of a processing strategy of the target service, competition simulation information corresponding to the target service, data fluctuation situation information of a first preset number of service traffic data samples copied from the target service, and strategy information of a competitor of a preset commodity corresponding to the target service; The training module performs model training on the service estimation model based on the historical service data, the service prediction result sample and the service processing data sample, to obtain a trained service estimation model.

[0105] In the embodiments of the present specification, the target service agent is a service agent for determining the selling price of a commodity; the service traffic data includes one or more of data of various requests for a preset commodity, search information input by a user and related to the preset commodity, and push information returned for the search information; the service estimation model is a model for performing revenue estimation processing based on the selling price of a commodity determined by the target service agent; the historical service data includes one or more of historical field traffic data of the target service, historical selling price ranking information corresponding to the target service, features of a preset commodity, features of a user, traffic features of a same-model or same-family commodity, and sales features of a same-model or same-family commodity; and the competition simulation information corresponding to the target service includes competition bidding simulation information corresponding to the target service.

[0106] In the embodiments of the present specification, the copied service traffic data includes a plurality of, the competition simulation module 1204, for any two adjacent service traffic data in the plurality of service traffic data, for the service traffic data arranged in front, taking the target service agent as a leader and a preset benchmark prediction model as a follower, determining competition situation simulation information corresponding to the target service based on historical service change trend information in the target service and the service prediction result through a preset dynamic game model; and the agent evaluation module 1204, based on the competition situation simulation information corresponding to the target service and the service prediction result, evaluates the target service agent needed to be deployed into the target service through a preset service estimation model, to obtain an evaluation result corresponding to the service traffic data arranged in front. The competition simulation module 1203, for the arranged later service traffic data, takes the target service agent as a leader and a preset benchmark prediction model as a follower, determines the competition situation simulation information corresponding to the target service based on the historical service change trend information in the target service, the service prediction result and the evaluation result corresponding to the arranged earlier service traffic data through a preset dynamic game model; the agent evaluation module 1204, based on the competition situation simulation information corresponding to the target service and the service prediction result, evaluates the target service agent needed to be deployed to the target service through a preset service estimation model, and obtains the evaluation result corresponding to the arranged later service traffic data; The agent evaluation module 1204 obtains the evaluation result corresponding to each service traffic data in the plurality of service traffic data based on the above processing mode, and determines the evaluation result of evaluating the target service agent needed to be deployed to the target service based on the evaluation result corresponding to each service traffic data in the plurality of service traffic data.

[0107] For the convenience of description, the above device is described as various modules or units respectively described in function. Of course, in the implementation of one or more embodiments of the present specification, the functions of each module or unit can be implemented in the same software and / or hardware, or the modules implementing the same function can be combined to realize the combination of a plurality of sub-modules or sub-units. The above described device embodiment is only illustrative, and the division of each module or unit is only a logical function division. In actual implementation, another division mode can be used, for example, a plurality of units or modules can be combined or integrated into another system, or some features can be ignored or not executed, etc.

[0108] The embodiment of the present specification provides an evaluation device of a business agent. By receiving an evaluation request of a target business agent, business traffic data is copied from a target business to be deployed by the target business agent. Based on the business traffic data, the target business agent is used for business prediction processing to obtain a corresponding business prediction result. Then, the target business agent can be used as a leader, and a preset benchmark prediction model can be used as a follower. Based on historical business trend information and the business prediction result in the target business, a preset dynamic game model is used to determine the corresponding competition situation simulation information of the target business. Finally, based on the corresponding competition situation simulation information and the business prediction result of the target business, a preset business estimation model is used to evaluate the target business agent to be deployed in the target business, and a corresponding evaluation result is obtained. In this way, by constructing a highly simulated virtual environment, the transaction environment of a certain time length can be restored in a short time (minute-hour level) for intelligent models (such as pricing models), hosting strategies and recommendation algorithms. The algorithm effect is comprehensively tested and evaluated. This sandbox environment can simulate various conventional or extreme trading conditions, user behaviors and competition situations, so as to quickly verify the effectiveness and risk of the intelligent agent in a safe and controllable environment, and drive the iterative optimization of the algorithm model. This method not only improves the accuracy and robustness of the model, but also significantly reduces the risk and cost of testing in the real environment through early estimation and verification.

[0109] The above is the evaluation device of the business agent provided by the embodiment of the present specification. Based on the same idea, the embodiment of the present specification also provides an evaluation device of a business agent, as shown in Figure 13

[0110] The evaluation device of the business agent can be a terminal device or a server provided in the above embodiment.

[0111] ​The business agent evaluation device can vary widely in configuration and performance, and can include a communications interface 1302, a user interface 1304, a processor 1306, and a data store 1308, which are communicatively coupled via a system bus, network, or other connection mechanism 1310. The communications interface 1302 enables the business agent evaluation device 1300 to communicate with other devices, access networks, and transport networks via analog or digital modulation. For example, the communications interface 1302 can include a chipset and antenna for wireless communication with a radio access network or access point. In addition, the communications interface 1302 can be a wired interface such as an Ethernet, Token Ring, or USB port, or a wireless interface such as Wifi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communications interface 1302 can support other forms of physical layer interface and standard or proprietary communication protocols. The communications interface 1302 can also include multiple physical communication interfaces, such as a Wifi interface, a Bluetooth interface, and a wide-area wireless interface.

[0112] The user interface 1304 includes receiving user input and providing output to a user. Thus, the user interface 1304 can include input components such as a keypad, keyboard, touch- sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which can be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other known or future developed equivalent devices. The user interface 1304 can also generate audible output through a speaker, speaker jack, audio output port, audio output device, headphones, and other known or future developed equivalent devices. In some embodiments, the user interface 1304 can include software, circuitry, or other forms of logic that enable the transmission of data to and from external user input / output devices. Additionally or alternatively, the business agent evaluation device 1300 can support remote access from other devices via the communications interface 1302 or another physical interface (not shown). The user interface 1304 can be configured to receive user input, the location and movement of which can be indicated by an indicator or cursor as described herein. The user interface 1304 can also be configured as a display device for rendering or displaying a text snippet.

[0113] The processor 1306 can include one or more general processors and / or dedicated processors.

[0114] The data store 1308 can include one or more volatile and / or non-volatile storage components, and can be entirely or partially integral with the processor 1306. The data store 1308 can include removable and / or non-removable storage components.

[0115] The processor 1306 is capable of executing program instructions stored in the data storage 1308 (e.g., compiled or interpreted program logic and / or machine code) to implement various functionality described herein. The data storage 1308 can include a non-transitory computer- readable medium having stored thereon program instructions that, when executed by the business agent evaluator device 1300, enable the business agent evaluator device 1300 to perform any of the methods, processes, or functions disclosed in the specification and / or drawings. Execution of the program instructions 1318 by the processor 1306 can cause the processor 1306 to utilize the data 1312.

[0116] For example, the program instructions 1318 can include an operating system 1322 (e.g., an operating system kernel, device drivers, and / or other modules) installed on the business agent evaluator device 1300, as well as one or more application programs 1320 (e.g., a browser, a social application, or a game application). Similarly, the data 1312 can include operating system data 1316 and application data 1314. The operating system data 1316 is primarily accessible to the operating system 1322, while the application data 1314 is primarily accessible to the one or more application programs 1320. The application data 1314 can be located in a file system that is visible or hidden to a user of the business agent evaluator device 1300.

[0117] The application programs 1320 can communicate with the operating system 1312 through one or more application programming interfaces (APIs). These APIs facilitate the application programs 1320 reading and / or writing to the application data 1314, communicating or receiving information via the communication interface 1302, receiving or displaying information on the user interface 1304, and the like.

[0118] In some terminology, the application programs 1320 can be referred to simply as “apps.” Furthermore, the application programs 1320 can be downloaded to the business agent evaluator device 1300 through one or more online application stores or application markets. However, the application programs can also be installed on the business agent evaluator device 1300 through other means, such as through a web browser or a physical interface (e.g., a USB port) on the business agent evaluator device 1300.

[0119] In particular embodiments, the business agent evaluator device 1300 includes a data storage 1308, and one or more program instructions 1318 stored in the data storage 1308, and configured to be executed by one or more processors, the one or more program instructions 1318 comprising computer-executable instructions for performing: receive an evaluation request for a target service agent, the target service agent being configured to perform a preset service prediction process on a target service to be deployed by the target service agent; copy service traffic data from the target service to be deployed by the target service agent, and perform a service prediction process on the target service agent based on the service traffic data to obtain a corresponding service prediction result; take the target service agent as a leader and a preset benchmark prediction model as a follower, determine corresponding competition situation simulation information of the target service based on historical service change trend information of the target service and the service prediction result through a preset dynamic game model; perform evaluation on the target service agent to be deployed in the target service based on the corresponding competition situation simulation information of the target service and the service prediction result through a preset service estimation model to obtain a corresponding evaluation result.

[0120] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the evaluation device embodiment of the service agent, the description is relatively simple because it is basically similar to the method embodiment, and the relevant parts can be referred to the part of the method embodiment.

[0121] The embodiment of the specification provides a service intelligent agent evaluation device, which copies service traffic data from a target service to which a target service intelligent agent needs to be deployed by receiving an evaluation request for the target service intelligent agent, and obtains corresponding service prediction results by performing service prediction processing on the target service intelligent agent based on the service traffic data. Then, the target service intelligent agent can be taken as a leader, and a preset benchmark prediction model can be taken as a follower. Based on historical service change trend information in the target service and the service prediction results, competitive situation simulation information corresponding to the target service is determined through a preset dynamic game model. Finally, the target service intelligent agent needed to be deployed to the target service can be evaluated based on the competitive situation simulation information corresponding to the target service and the service prediction results through a preset service estimation model, and corresponding evaluation results are obtained. In this way, by constructing a highly simulated virtual environment, the algorithm effect can be comprehensively tested and evaluated in a transaction environment restored for a certain length of time (minute-hour level) for intelligent models (such as pricing models), hosting strategies and recommendation algorithms. The sandbox environment can simulate various conventional or extreme transaction conditions, user behaviors and competitive situations, so as to quickly verify the effectiveness and risk of the intelligent agent in a safe and controllable environment, and drive the iteration and optimization of the algorithm model. This method can not only improve the accuracy and robustness of the model, but also significantly reduce the risk and cost of testing in a real environment through early estimation and verification.

[0122] Further, based on the above Figures 1 to 11 One or more embodiments of the specification also provide a storage medium for storing computer executable instruction information. In a specific embodiment, the storage medium can be a U disk, an optical disk, a hard disk, etc. The computer executable instruction information stored in the storage medium can implement the following processes when executed by a processor. Receiving an evaluation request for a target service intelligent agent, the target service intelligent agent being configured to perform preset service prediction processing on a target service to which the target service intelligent agent needs to be deployed. Copying service traffic data from the target service to which the target service intelligent agent needs to be deployed, and performing service prediction processing on the target service intelligent agent based on the service traffic data to obtain corresponding service prediction results. Taking the target service intelligent agent as a leader, and taking a preset benchmark prediction model as a follower. Based on historical service change trend information in the target service and the service prediction results, competitive situation simulation information corresponding to the target service is determined through a preset dynamic game model. Based on the competitive situation simulation information corresponding to the target service and the service prediction results, the target service intelligent agent needed to be deployed to the target service is evaluated through a preset service estimation model, and corresponding evaluation results are obtained.

[0123] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the above-mentioned storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0124] The embodiment of the specification provides a storage medium, by receiving an evaluation request for a target business agent, copying business traffic data from a target business which needs to be deployed by the target business agent, and based on the business traffic data, performing business prediction processing through the target business agent to obtain a corresponding business prediction result, then taking the target business agent as a leader and a preset benchmark prediction model as a follower, determining the corresponding competition situation simulation information of the target business through a preset dynamic game model based on the historical business change trend information and the business prediction result in the target business, finally, the target business agent which needs to be deployed to the target business can be evaluated based on the corresponding competition situation simulation information and the business prediction result of the target business through a preset business estimation model to obtain a corresponding evaluation result. In this way, by constructing a highly simulated virtual environment, the algorithm effect can be fully tested and evaluated in a certain length of transaction environment (minute-hour level) for intelligent models (such as pricing models), hosting strategies and recommendation algorithms in a short time (minute-hour level). The sandbox environment can simulate various conventional or extreme trading conditions, user behaviors and competition situations, so as to quickly verify the effectiveness and risk of the intelligent agent in a safe and controllable environment, and drive the iterative optimization of the algorithm model. This way not only can improve the accuracy and robustness of the model, but also can significantly reduce the risk and cost of testing in the real environment through early estimation and verification.

[0125] Further, based on the above Figures 1 to 11 One or more embodiments of the specification also provide a computer program product, including a computer program, and the computer program in the computer program product can implement the following flow when executed by a processor. Receiving an evaluation request for a target business agent, the target business agent being used for performing a preset business prediction processing on a target business which needs to be deployed by the target business agent; Copying business traffic data from a target business which needs to be deployed by the target business agent, and based on the business traffic data, performing business prediction processing through the target business agent to obtain a corresponding business prediction result; Take the target business agent as the leader, and take the preset benchmark prediction model as the follower, based on the historical business change trend information in the target business and the business prediction result, determine the corresponding competitive situation simulation information of the target business through the preset dynamic game model; Based on the competitive situation simulation information corresponding to the target business and the business prediction result, the target business agent needed to be deployed in the target business is evaluated through the preset business estimation model, and a corresponding evaluation result is obtained.

[0126] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the above-mentioned computer program product embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0127] The embodiment of the specification provides a computer program product, by receiving the evaluation request of the target business agent, copying the business traffic data from the target business which needs to be deployed by the target business agent, and based on the business traffic data, performing business prediction processing through the target business agent to obtain a corresponding business prediction result, then, taking the target business agent as the leader, and taking the preset benchmark prediction model as the follower, based on the historical business change trend information in the target business and the business prediction result, determining the competitive situation simulation information corresponding to the target business through the preset dynamic game model, finally, based on the competitive situation simulation information corresponding to the target business and the business prediction result, the target business agent needed to be deployed in the target business is evaluated through the preset business estimation model, and a corresponding evaluation result is obtained. In this way, by constructing a highly simulated virtual environment, the transaction environment of a certain time length can be restored in a short time (minute-hour level) for intelligent models (such as pricing models), hosting strategies and recommendation algorithms, and the algorithm effect can be comprehensively tested and evaluated. This sandbox environment can simulate various conventional or extreme trading conditions, user behaviors and competitive situations, so as to quickly verify the effectiveness and risk of the intelligent agent in a safe and controllable environment, and drive the iterative optimization of the algorithm model. This way not only can improve the accuracy and robustness of the model, but also can significantly reduce the risk and cost of testing in the real environment through early estimation and verification.

[0128] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. Moreover, although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible execution order among many steps and does not represent the only execution order. Therefore, when method steps are involved in the claims, adjustments to the order of those steps, or parallelism between steps, are also within the scope of protection of the claims.

[0129] In the 1990s, it was possible to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has advanced, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into a hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it themselves, without having to ask a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating an integrated circuit chip, this programming is now mostly implemented using "logic compiler" software, which is similar to the software compiler used when developing a program, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit that implements the logical method flow can be easily obtained.

[0130] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the microprocessor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to implementing the controller in pure computer readable program code, it is possible to implement the same functionality in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Such a controller can therefore be considered to be a hardware component, and the means included therein for implementing the various functions can also be considered to be structures within the hardware component. Alternatively, or even additionally, the means for implementing the various functions can be considered to be both a software module implementing the method and a structure within the hardware component.

[0131] The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0132] For the sake of description, the above apparatuses are described in functional division and are described respectively. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware when implementing one or more embodiments of the present specification.

[0133] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of the present specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] The embodiments of the present specification are described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present specification. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable electronic devices to produce a machine, so that the instructions executed by the computer or other programmable electronic devices generate a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions of one or more flows and / or blocks Figure 1 The functions of one or more flows and / or blocks

[0135] These computer program instructions can also be stored in a computer readable storage medium that can direct the computer or other programmable electronic devices to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions of one or more flows and / or blocks Figure 1 The functions of one or more flows and / or blocks

[0136] These computer program instructions can also be loaded into a computer or other programmable electronic devices, so that a series of operation steps are performed on the computer or other programmable electronic devices to produce a computer implemented process, so that the instructions executed on the computer or other programmable electronic devices provide steps for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions of one or more flows and / or blocks Figure 1 The functions of one or more flows and / or blocks

[0137] In a typical configuration, the computing device includes one or more processors (CPU), input / output interface, network interface and memory.

[0138] The memory can include non-persistent memory in the computer readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer readable media.

[0139] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0140] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical or equivalent elements in the process, method, article or device comprising the element. In addition, "one", "a" and "the" do not necessarily refer to the singular, but also include the plural. The ordinal numbers first, second, etc. do not necessarily indicate the order, but are often used for the purpose of distinguishing objects. For example, the first server and the second server usually refer to two servers, which are expressed as the first server and the second server in order to distinguish the two servers. However, the two servers may also be the same server at times. Moreover, in the present specification, unless otherwise specified, "receiving and sending of data" is not necessarily direct receiving and sending, but can be indirect receiving and sending (i.e. indirect receiving and sending through one or more subjects). Similarly, in the present specification, unless otherwise specified, the association relationship between structures can be a direct association relationship or an indirect association relationship.

[0141] In addition, as used in the specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a" entity includes reference to one or more of such entities. In addition, as used in this specification and the appended claims, the term "or" is generally employed in its sense of "and / or" unless the context clearly dictates otherwise.

[0142] Persons skilled in the art will understand that embodiments of the present specification can be provided as methods, systems or computer program products. Accordingly, one or more embodiments of the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present specification can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0143] One or more embodiments of the present specification can be described in the general context of computer-executable instructions, such as program modules, being executed by computers. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. One or more embodiments of the present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0144] Embodiments of the present specification are described with reference to the attached drawings, wherein like numerals indicate like elements throughout the several figures. The embodiments of the present specification are described with reference to the attached figures, wherein like numerals indicate like elements throughout the several figures. Each embodiment of the present specification is described in progressive manner, and the same or similar parts among the embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments.

[0145] The above only describes the embodiments of the present specification and is not intended to limit the present specification. The present specification can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for evaluating a business intelligence agent, the method comprising: Receive an evaluation request for a target business intelligence agent, wherein the target business intelligence agent is used to perform preset business prediction processing on the target business that the target business intelligence agent needs to deploy; Copy the business traffic data from the target business that the target business agent needs to deploy, and based on the business traffic data, perform business prediction processing through the target business agent to obtain the corresponding business prediction results. Using the target business intelligent agent as the leader and a preset benchmark prediction model as the follower, based on the historical business change trend information in the target business and the business prediction results, the competitive situation simulation information corresponding to the target business is determined through a preset dynamic game model. Based on the competitive situation simulation information corresponding to the target business and the business prediction results, the target business agent to be deployed in the target business is evaluated through a preset business prediction model to obtain the corresponding evaluation results.

2. The method according to claim 1, wherein the step of performing business prediction processing based on the business traffic data through the target business intelligence agent to obtain the corresponding business prediction result includes: Construct a shadow link for evaluating the target business agent, the shadow link being isolated from the processing link of the target business; Under the shadow link, the following processing is performed: The business traffic data is tagged to identify it as traffic data used to evaluate the target business intelligent agent, thereby obtaining the tag information corresponding to the business traffic data; Preprocessing is performed on the service traffic data with tagged information to obtain preprocessed service traffic data. The preprocessing includes filtering and / or data modification. Based on the preprocessed business traffic data, the target business agent performs business prediction processing to obtain the corresponding business prediction results, and stores the business prediction results in the shadow database corresponding to the shadow link.

3. The method according to claim 2, further comprising: Under the shadow link, the business processing corresponding to the target business is executed based on the business prediction result to obtain the corresponding business processing data; The business processing data is stored in the shadow database; Based on the competitive situation simulation information corresponding to the target business and the business prediction results, the target business agent to be deployed in the target business is evaluated using a preset business prediction model to obtain corresponding evaluation results, including: The business prediction results and business processing data are obtained from the shadow database, and features are extracted from the business prediction results and business processing data to obtain corresponding data features. Based on the competitive situation simulation information and data characteristics corresponding to the target business, the target business agent to be deployed in the target business is evaluated through a preset business prediction model, and the corresponding evaluation results are obtained.

4. The method according to any one of claims 1-3, wherein the step of determining the competitive situation simulation information corresponding to the target business through a preset dynamic game model, based on the historical business change trend information in the target business and the business prediction results, using the target business intelligent agent as the leader and a preset benchmark prediction model as the follower, includes: Using the target business intelligence agent as the leader and a preset benchmark prediction model as the follower, the decision range of the leader and the decision range of the follower are obtained. The decision information of the leader is determined based on the decision scope of the leader and the business prediction results, and the business prediction results corresponding to the leader are generated by the target business agent based on the decision information of the leader. Based on the business forecast results corresponding to the decision information and the historical business change trend information in the target business, the current competitive situation information is obtained through the benchmark prediction model, and a follower reaction function is constructed based on the current competitive situation information. Based on the business prediction results corresponding to the decision information, the business prediction results corresponding to the followers are generated through the follower reaction function; Based on the business forecast results corresponding to the dominant player and the business forecast results corresponding to the follower, the competitive situation simulation information corresponding to the target business is determined.

5. The method according to claim 2, further comprising: Obtain sample data for training the business prediction model. The sample data includes historical business data related to the target business and business prediction result samples obtained by business prediction processing through the target business intelligent agent. Based on the different functional information of the business prediction model, a model architecture matching each functional information is obtained from a preset business modeling database, and the business prediction model is constructed based on the obtained model architecture. Based on the historical business data and the business prediction result samples, the business prediction model is trained to obtain the trained business prediction model.

6. The method according to claim 5, further comprising: Obtain business processing data samples obtained by performing business processing corresponding to the target business based on the business prediction result sample under the shadow link. The business processing data samples include simulation information of the processing strategy of the target business, competitive simulation information corresponding to the target business, data fluctuation information of a first preset number of business traffic data samples copied from the target business, and strategy information of competitors of the preset goods corresponding to the target business. The step of training the business prediction model based on the historical business data and the business prediction result samples to obtain the trained business prediction model includes: Based on the historical business data, the business prediction result sample, and the business processing data sample, the business prediction model is trained to obtain the trained business prediction model.

7. The method according to claim 6, wherein the target business agent is a business agent used to determine the selling price of a product; the business traffic data includes one or more of the following: data of various requests for a preset product, search information related to the preset product input by the user, and push information returned for the search information; the business prediction model is a model used to perform revenue prediction processing based on the selling price of the product determined by the target business agent; the historical business data includes one or more of the following: historical field traffic data of the target business, historical selling price ranking information corresponding to the target business, features of the preset product, features of the user, traffic features corresponding to the same or similar products, and sales features corresponding to the same or similar products; the competitive simulation information corresponding to the target business includes bidding simulation information corresponding to the target business.

8. The method according to claim 4, wherein the copied business traffic data includes multiple components, wherein the target business intelligent agent is the leader and a preset benchmark prediction model is the follower, and the competitive situation simulation information corresponding to the target business is determined by a preset dynamic game model based on the historical business change trend information in the target business and the business prediction results; Based on the competitive situation simulation information corresponding to the target service and the service prediction results, the target service agent to be deployed in the target service is evaluated using a preset service prediction model to obtain corresponding evaluation results, including: For any two adjacent service traffic data in multiple service traffic data sets, for the service traffic data that appears first, the target service agent is used as the leader, and a preset benchmark prediction model is used as the follower. Based on the historical service change trend information in the target service and the service prediction results, a preset dynamic game model is used to determine the competitive situation simulation information corresponding to the target service. Based on the competitive situation simulation information corresponding to the target service and the service prediction results, a preset service prediction model is used to evaluate the target service agent that needs to be deployed in the target service, and the evaluation results corresponding to the service traffic data that appears first are obtained. For the business traffic data ranked later, the target business agent acts as the leader, and a preset benchmark prediction model acts as the follower. Based on the historical business change trend information in the target business, the business prediction results, and the evaluation results corresponding to the business traffic data ranked earlier, a preset dynamic game model is used to determine the competitive situation simulation information corresponding to the target business. Based on the competitive situation simulation information corresponding to the target business and the business prediction results, a preset business estimation model is used to evaluate the target business agent that needs to be deployed in the target business, and the evaluation results corresponding to the business traffic data ranked later are obtained. Based on the above processing method, the evaluation results corresponding to each business traffic data in multiple business traffic data are obtained respectively. Based on the evaluation results corresponding to each business traffic data in multiple business traffic data, the evaluation results for evaluating the target business intelligent agent that needs to be deployed in the target business are determined.

9. An evaluation device for a business intelligence agent, the device comprising: The evaluation request module receives evaluation requests for the target business intelligence agent, which is used to perform preset business prediction processing on the target business that needs to be deployed by the target business intelligence agent. The business prediction module copies business traffic data from the target business that the target business agent needs to deploy, and performs business prediction processing through the target business agent based on the business traffic data to obtain the corresponding business prediction results. The competition simulation module takes the target business intelligent agent as the leader and a preset benchmark prediction model as the follower. Based on the historical business change trend information in the target business and the business prediction results, it determines the competition situation simulation information corresponding to the target business through a preset dynamic game model. The intelligent agent evaluation module evaluates the intelligent agents of the target business that need to be deployed in the target business based on the competitive situation simulation information corresponding to the target business and the business prediction results, and obtains the corresponding evaluation results.

10. An evaluation device for a business intelligence agent, the evaluation device for the business intelligence agent comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to: Receive an evaluation request for a target business intelligence agent, wherein the target business intelligence agent is used to perform preset business prediction processing on the target business that the target business intelligence agent needs to deploy; Copy the business traffic data from the target business that the target business agent needs to deploy, and based on the business traffic data, perform business prediction processing through the target business agent to obtain the corresponding business prediction results. Using the target business intelligent agent as the leader and a preset benchmark prediction model as the follower, based on the historical business change trend information in the target business and the business prediction results, the competitive situation simulation information corresponding to the target business is determined through a preset dynamic game model. Based on the competitive situation simulation information corresponding to the target business and the business prediction results, the target business agent to be deployed in the target business is evaluated through a preset business prediction model to obtain the corresponding evaluation results.

Images