Data processing method, device, readable storage medium and electronic device

Abstract

The embodiments of the present invention disclose a data processing method, device, readable storage medium and electronic device, which determine a target model used online and at least one corresponding offline backup model, and determine the performance difference value between each backup model and the target model. When the performance difference value meets the first adjustment condition, first adjustment information including a backup model identifier is generated, which is used to adjust the target model in an updated manner. Target adjustment information is determined based on the first adjustment information to adjust the target model. The embodiments of the present invention monitor the model performance by determining the offline backup model corresponding to the target model, and automatically determine the corresponding adjustment method when the performance of the backup model is better than the target model, so as to adjust the target model through optimization or replacement, improve the online model performance, and realize intelligent model monitoring, maintenance and management.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method, device, readable storage medium and electronic device. Background Art

[0002] Currently, machine learning models are widely used in various fields. How to cope with large-scale and large-volume model maintenance in multiple categories and business scenarios is a time-consuming, labor-intensive and cost-intensive problem. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a data processing method, device, readable storage medium and electronic device, aiming to realize intelligent model monitoring, maintenance and management.

[0004] In a first aspect, an embodiment of the present invention discloses a data processing method, the method comprising:

[0005] Determining a target model and at least one corresponding backup model;

[0006] Determine a performance difference value between each of the backup models and the target model, wherein the performance difference value is used to characterize a performance difference between the corresponding backup model and the target model;

[0007] In response to the performance difference value satisfying a first adjustment condition, generating first adjustment information including a backup model identifier for adjusting the target model in an updated manner;

[0008] Target adjustment information is determined according to the first adjustment information, so as to adjust the target model by using the target adjustment information.

[0009] In a second aspect, an embodiment of the present invention discloses a data processing device, the device comprising:

[0010] A model determination module, used to determine a target model and at least one corresponding backup model;

[0011] A model monitoring module, used to determine a performance difference value between each of the backup models and the target model, wherein the performance difference value is used to characterize a performance difference between the corresponding backup model and the target model;

[0012] an information generating module, configured to generate first adjustment information including a spare model identifier in response to the performance difference satisfying a first adjustment condition, for adjusting the target model in an updating manner;

[0013] The model adjustment module is used to determine target adjustment information according to the first adjustment information, so as to adjust the target model according to the target adjustment information.

[0014] In a third aspect, an embodiment of the present invention discloses a computer-readable storage medium for storing computer program instructions, wherein the computer program instructions implement the method described in the first aspect when executed by a processor.

[0015] In a fourth aspect, an embodiment of the present invention discloses an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0016] The embodiment of the present invention determines a target model used online and at least one corresponding offline backup model, and determines the performance difference value between each backup model and the target model. When the performance difference value meets the first adjustment condition, first adjustment information including a backup model identifier is generated for adjusting the target model in an updated manner. Target adjustment information is determined based on the first adjustment information to adjust the target model. The embodiment of the present invention monitors model performance by determining an offline backup model corresponding to the target model, and automatically determines a corresponding adjustment method when the performance of the backup model is better than the target model, thereby adjusting the target model through optimization or replacement, improving the online model performance, and realizing intelligent model monitoring, maintenance and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0018] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of a target model and a backup model according to an embodiment of the present invention;

[0020] Figure 3 A flowchart of a process for determining first adjustment information according to an embodiment of the present invention;

[0021] Figure 4 A flowchart of a process for determining second adjustment information according to an embodiment of the present invention;

[0022] Figure 5 A flowchart of a process for determining target adjustment information according to an embodiment of the present invention;

[0023] Figure 6 is a schematic diagram of a data processing device according to an embodiment of the present invention;

[0024] Figure 7 Schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, flows, components and circuits are not described in detail.

[0026] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0027] Unless the context clearly requires otherwise, words such as “include”, “including” and the like in the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0028] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0029] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention, as shown in Figure 1 As shown, the data processing method comprises the following steps:

[0030] Step S100: determine a target model and at least one corresponding backup model.

[0031] Specifically, the data processing method of the embodiment of the present invention is implemented through a server, and the server can be a single server or a server cluster configured in a distributed manner. The server can run multiple online models through a background program, and store at least one offline backup model corresponding to each online model through an offline storage medium such as a local database or a connected remote database and a cloud database. Each of the backup models plays the same role as the corresponding online model and is used to replace the corresponding online model when certain conditions are met. Furthermore, the server determines any one model as the target model from the multiple online models running in the background, and determines at least one offline stored backup model corresponding to the target model from each offline model stored in the database.

[0032] Figure 2 FIG. 2 is a schematic diagram of a target model and a backup model according to an embodiment of the present invention. Figure 2As shown, the server determines the target model 20 in the model run by the background program, and determines four backup models 21 corresponding to the target model 20 in the offline storage medium, which are backup model 1, backup model 2, backup model 3 and backup model 4. In an embodiment of the present invention, the model run by the server background program is a business model of the server corresponding to the business, which can be applied to any field where a machine learning model needs to be applied. For example, when the server is a server in the field of online car-hailing, the business model run by the background program may include a travel duration estimation model for predicting the processing time of online car-hailing orders, an order quantity estimation model for predicting the number of online car-hailing orders within a period of time, and a path estimation model for predicting a path among multiple paths corresponding to online car-hailing orders. Further, the backup model 21 corresponding to the target model 20 has the same function as the target model 20. That is, when the target model 20 is used to predict the processing time of online car-hailing orders, the corresponding backup model 21 is also used to predict the processing time of online car-hailing orders. When the target model 20 is used to predict the number of online car-hailing orders within a period of time, the corresponding backup model 21 is also used to predict the number of online car-hailing orders within a period of time.

[0033] Step S200: Determine the performance difference value between each of the backup models and the target model.

[0034] Specifically, the server monitors the performance of the target model in real time during the process of the target model being run by the background program, so as to maintain and manage the target model by updating or optimizing when the performance of the target model does not meet the preset requirements. In an optional implementation of an embodiment of the present invention, the monitoring method of the server is to determine the performance difference value between each of the backup models and the target model, and the monitoring process may be to determine the performance difference value between the target model and each of the backup models according to a preset monitoring period, or to determine the performance difference value between the target model and each of the backup models according to a monitoring command received by the server. The performance difference value is used to characterize the performance difference between the corresponding backup model and the target model.

[0035] Furthermore, the process of determining the performance difference value between each of the backup models and the target model in the embodiment of the present invention further includes the following steps:

[0036] Step S210: Determine the first offline performance and the second offline performance corresponding to the target model and each of the backup models respectively according to a preset offline sample set.

[0037] Specifically, the offline storage medium of the server also stores an offline sample set corresponding to each of the target models, and the offline sample set includes at least one offline sample used as a model input and a corresponding offline output. Optionally, the offline sample set can be determined by obtaining online data of the target model input within a preset time period and the corresponding real results, wherein the input of the target model is used as the offline input and the corresponding real results are used as the offline output. Alternatively, the offline sample set can also be a plurality of offline samples and corresponding offline output sets determined by the server through offline simulation based on real data.

[0038] Take the target model of the embodiment of the present invention for determining the duration of an online car-hailing trip, where the corresponding input data is the driver information, passenger information, starting position and ending position, and the output is the corresponding trip duration as an example for explanation. The offline sample set can be determined based on the historical online car-hailing orders received by the server within one month before the current time, that is, the driver information, passenger information, starting position and ending position corresponding to each of the historical online car-hailing orders are used as offline samples, and the corresponding actual completion duration is used as the offline result to determine the offline sample set. Furthermore, the offline sample set can also be an offline simulation performed by the server based on the correspondence between historical online car-hailing order data and trip duration, and multiple virtual driver information, virtual passenger information, virtual starting position and virtual ending position are obtained as offline samples, and multiple virtual completion times that have a corresponding relationship with the offline samples are used as offline results to determine the offline sample set.

[0039] Furthermore, after determining the offline sample set, the server inputs the offline samples in the offline sample set into the target model in an offline test environment to obtain the corresponding target offline output. At the same time, the server also uses each offline sample in the offline sample set as the input of each backup sample to determine at least one backup offline output corresponding to each backup sample.

[0040] Furthermore, the server determines the first offline performance corresponding to the target model according to the offline output corresponding to each of the offline samples and the target offline output. At the same time, for each of the backup models, the server determines the corresponding second offline performance according to the offline output corresponding to each offline sample and the backup offline output. The first offline performance is used to characterize the performance index of the target model obtained after being tested by the offline sample set, and the second offline performance is used to characterize the performance index of the corresponding backup model obtained after being tested by the offline sample set, and the first offline performance and the second offline performance are used to characterize at least one of the same performance indicators, and the performance indicators can be ROC-AUC (Receiver Operating Characteristic-Area Under Curve), PRC-AUC (Precision Recall Curve-Area Under Curve), accuracy, recall rate, etc., which are used to evaluate model performance.

[0041] Step S220: Determine a corresponding performance difference value according to the difference between the first offline performance and each of the second offline performances.

[0042] Specifically, the first offline performance and the second offline performance can be directly represented by numerical values, or converted into corresponding numerical values. The server can respectively calculate the difference between the numerical value corresponding to the first offline performance and the numerical value of each second offline performance, and use the difference as the performance difference value for characterizing the performance difference between the target model and the corresponding backup model.

[0043] Take the example that the first offline performance is used to characterize the accuracy corresponding to the target model, and each of the second offline performances is used to characterize the accuracy of the corresponding backup model. When the target model has four backup models, and the first offline performance corresponding to the target model is 0.75, and the second offline performances corresponding to backup model 1, backup model 2, backup model 3 and backup model 4 are 0.66, 0.73, 0.70 and 0.78 respectively, the performance difference values ​​corresponding to each of the backup models are 0.09, 0.02, 0.05 and -0.03 respectively. Since the higher the accuracy of the model, the better the performance, when the corresponding performance difference value is positive, it is judged that the performance of the target model is better than the corresponding backup model; when the corresponding performance difference value is negative, it is judged that the performance of the target model is worse than the corresponding backup model. In other words, among the four backup models corresponding to the above target model, only the performance of backup model 4 is better than the target model.

[0044] Step S300: In response to the performance difference value satisfying a first adjustment condition, generating first adjustment information including a backup model identifier.

[0045] Specifically, after determining the performance difference value between each of the backup models and the target model, the server determines whether the target model needs to be adjusted in the form of an update according to each of the performance difference values, and generates corresponding first adjustment information if it needs to be adjusted in the form of an update. The first adjustment information includes a backup model identifier for characterizing the backup model for model update.

[0046] In an embodiment of the present invention, the server determines whether the target model needs to be adjusted by judging whether each of the performance difference values ​​satisfies the first adjustment condition. Since the performance difference value is determined based on the first offline performance corresponding to the target model and the second offline model corresponding to each backup model, the larger the performance difference value, the better the performance of the target model is compared with the backup model, and the smaller the performance difference value, the worse the performance of the target model is compared with the backup model. The first adjustment condition can be determined according to the size of the performance difference value, for example, determining whether the performance difference value corresponding to each of the backup models is less than a preset first threshold, so as to judge whether the target model needs to be adjusted according to the difference between each of the performance difference values ​​and the first threshold. Optionally, the first threshold is a value less than or equal to 0. When there is only one performance difference value less than the first threshold, the first adjustment information including the backup model identifier corresponding to the performance difference value is generated. When there is a performance difference value greater than one less than the first threshold, the sizes of the performance difference values ​​are compared, and the first adjustment information including the backup model identifier with the smallest performance difference value is determined.

[0047] Still taking the example that the target model has four backup models, and the first offline performance corresponding to the target model is 0.75, and the second offline performance corresponding to backup model 1, backup model 2, backup model 3 and backup model 4 are 0.66, 0.73, 0.70 and 0.78 respectively. The performance difference values ​​corresponding to each of the backup models are 0.09, 0.02, 0.05 and -0.03 respectively. When the first threshold is 0, the backup model 4 is determined to be the backup model for replacing the target model, and the first adjustment information including the backup model identifier corresponding to the backup model 4 is determined.

[0048] Figure 3 FIG. 1 is a flow chart of a process for determining first adjustment information according to an embodiment of the present invention. Figure 3 As shown, in an optional implementation of the embodiment of the present invention, the server determines whether the target model needs to be adjusted according to the performance difference value between the target model and the backup model, and determines the corresponding first adjustment information when adjustment is required. The process of determining the first adjustment information includes the following steps:

[0049] S30: Determine a first offline performance for characterizing the performance of the target model based on the offline sample set.

[0050] S31, determining a second offline performance for characterizing the performance of each backup model corresponding to the target model based on the offline sample set. Step S31 and step S30 may be performed simultaneously.

[0051] S32, determining the performance difference value between the corresponding standby model and the target model according to the first offline performance and each second offline performance, wherein the larger the performance difference value, the better the target model performance is than the standby model, and the smaller the performance difference value, the worse the target model performance is than the standby model.

[0052] S33: Determine whether the performance difference value is less than a preset first threshold, where the first threshold is a value less than or equal to 0.

[0053] S34. When there is a performance difference value less than a preset first threshold, the server determines that the target model needs to be updated and adjusted, and generates a first adjustment signal. When there is only one performance difference value less than the first threshold, a first adjustment signal including a standby model identifier for characterizing the standby model corresponding to the performance difference value is determined. When there is more than one performance difference value less than the first threshold, a first adjustment signal including a standby model identifier corresponding to the standby model with the smallest performance difference value is determined.

[0054] S35: When there is no performance difference value less than the preset first threshold, the server determines that the target model does not need to be updated and adjusted.

[0055] Furthermore, while determining whether the target model needs to be adjusted by updating, the server can also determine whether the target model needs to be adjusted by optimizing. Specifically, the server also sets a second adjustment condition for monitoring whether the target model needs to be optimized, and generates second adjustment information for adjusting the target model in an optimized manner when the second adjustment condition is triggered. Optionally, the server can monitor the target model from multiple dimensions such as data, performance, and time. When one of the dimensions meets the preset conditions, it is considered that the second adjustment condition is triggered, and the target model needs to be optimized, and the corresponding second adjustment information is generated.

[0056] In an optional implementation of the embodiment of the present invention, the second adjustment condition can be triggered based on the performance fluctuation of the target model, that is, the target model triggers the second adjustment condition when the performance decreases, and generates corresponding second adjustment information. The process of generating the second adjustment information by monitoring the performance of the target model includes the following steps:

[0057] Step S300': determine the performance characteristic value of the target model.

[0058] Specifically, the server determines an online sample set and historical performance parameters determined by the target model, wherein the online sample set includes at least one online sample for inputting the target model and a corresponding real result, and the historical performance parameters are used to characterize the historical performance indicators of the target model, which can be determined based on all or part of the online samples of the historical input of the target model and the historical prediction results and historical real results of the model output, and the part of the online samples are online samples obtained within a time period before a preset time period of the current moment.

[0059] Further, the server determines the current performance parameters of the target model based on the online sample set. The online sample set can be determined by obtaining the online data of the target model input within a preset time period and the corresponding real results, wherein the input of the target model is used as an online sample, and the online sample set is determined according to each of the online samples and the corresponding real results. The server inputs each of the online samples into a preset target model to obtain a corresponding prediction result, and determines the current performance parameters of the target model according to the prediction result and the real result of each of the online samples. The current performance parameter is used to characterize the current performance index of the target model. The current performance parameter and the historical performance parameter are used to characterize at least one of the same performance indicators, and the performance indicator can be an indicator for evaluating model performance, such as ROC-AUC (Receiver Operating Characteristic-Area Under Curve), PRC-AUC (Precision Recall Curve-Area Under Curve), accuracy, recall rate, etc.

[0060] Take the target model of the embodiment of the present invention for determining the duration of an online car-hailing trip, the corresponding input data is the driver information, the passenger information, the starting location and the ending location, and the output is the corresponding trip duration as an example for explanation. The online sample set can be determined based on the historical online car-hailing orders received by the server within one month before the current time, that is, the driver information, passenger information, starting location and ending location corresponding to each of the historical online car-hailing orders are used as online samples, and the corresponding actual completion duration is used as the real result to determine the online sample set.

[0061] Furthermore, after determining the historical performance parameters and current performance parameters corresponding to the target model, the service determines the corresponding performance characteristic values ​​according to the historical performance parameters and the current performance parameters. The process of determining the performance characteristic values ​​may be to first determine the numerical representations corresponding to the historical performance parameters and the numerical representations corresponding to the current performance parameters, and then determine the corresponding performance characteristic values ​​by calculating the difference between the numerical representations corresponding to the historical performance parameters and the current performance parameters. In an embodiment of the present invention, the higher the historical performance parameters, the better the historical target model performance, and the higher the current performance parameters, the better the current target model performance. Therefore, the larger the performance characteristic value, the greater the degree of performance degradation of the target model.

[0062] Step S310 ′: in response to the performance characteristic value being greater than a second threshold, generating second adjustment information for adjusting the target model in an optimized manner.

[0063] Specifically, after determining each of the performance characteristic values, the server determines the relationship between the performance characteristic value and a preset second threshold value to determine whether to trigger the second adjustment condition. When the performance characteristic value is greater than the second threshold value, the server determines that the performance of the target model has declined to a large extent, and the target model needs to be optimized for performance, and generates corresponding second adjustment information. When the performance characteristic value is less than the second threshold value, the server determines that the performance of the target model has not declined to a large extent, and no performance optimization is required.

[0064] In an alternative implementation of the embodiment of the present invention, the second adjustment condition can be triggered based on the fluctuation of the input data of the target model, that is, the target model triggers the second adjustment condition when the data fluctuation is large, and generates corresponding second adjustment information. The process of generating the second adjustment information by monitoring the data corresponding to the target model includes the following steps:

[0065] Step S320': determine the data feature value corresponding to the target model.

[0066] Specifically, the server can also determine whether model optimization is needed by monitoring the fluctuation of data features corresponding to the target model. Among them, the monitoring process of the server can be to first determine the historical data and current data corresponding to the target model. The historical data corresponding to the target model is used to characterize the historical features of at least one parameter that has a preset relationship with the target model, and can be determined by obtaining all or part of the historical at least one parameter that has a preset relationship with the target model, and the partial parameter is a parameter obtained in a time period before the preset time period of the current moment. The current data is used to characterize the current features of at least one parameter that has a preset relationship with the target model, and can be determined by obtaining at least one parameter that has a preset relationship with the target model in a preset time period. The preset relationship between the parameter and the target model is a relationship that affects the performance of the target model, that is, the parameter can be an online sample of the input target model, the real output corresponding to the online sample, and other parameters preset by the server that have a corresponding relationship with the target model and affect the performance of the target parameter. Optionally, the data feature value can be determined by monitoring the distribution of production environment sample data, or by monitoring the production environment sample annotation data and annotation scheme, and by monitoring the quality of online environment feature data.

[0067] The target model of the embodiment of the present invention is used to determine the probability of negative reviews for online ride-hailing orders. The server can pre-set a parameter corresponding to the target model as the probability of negative reviews on the order day, and obtain the probability of negative reviews for each day within 30 days before the current date as current data. At the same time, the probability of negative reviews for each day within 60 days to 30 days before the current date is obtained as historical data.

[0068] Furthermore, the server determines a data characteristic value for characterizing the difference between the historical data and the current data based on the historical data and the current data. When the historical data and the current data have corresponding numerical values, the data characteristic value is determined by directly calculating the numerical difference corresponding to the historical data and the current data. The numerical values ​​corresponding to the historical data and the current data may be characteristic values ​​such as mean values ​​and variances. Optionally, when the historical data and the current data respectively include multiple numerical values, the data characteristic value may also be determined by determining the correlation coefficient between the curve corresponding to the historical data and the curve corresponding to the current data. The larger the data characteristic value, the greater the difference between the historical data and the current data, and the greater the impact on the performance of the target model.

[0069] Step S330': in response to the data characteristic value being greater than a third threshold, generating second adjustment information for adjusting the target model in an optimized manner.

[0070] Specifically, after determining each of the data feature values, the server determines the relationship between the data feature value and a preset third threshold value to determine whether to trigger the second adjustment condition. When the data feature value is greater than the third threshold value, the server determines that the performance of the target model will be greatly affected, and the target model needs to be optimized, and generates corresponding second adjustment information. When the data feature value is less than the third threshold value, the server determines that the performance of the target model has not declined to a large extent, and no performance optimization is required.

[0071] In another optional implementation of the embodiment of the present invention, the second adjustment condition can also be triggered based on a preset time period, that is, the second adjustment condition is periodically triggered according to the preset time period, and the corresponding second adjustment information is periodically generated. For example, when the preset time period is 1 month, the server generates the second adjustment information once a month for adjusting the target model in an optimized manner.

[0072] Figure 4 FIG. 1 is a flow chart of a process for determining the second adjustment information according to an embodiment of the present invention. Figure 4 As shown, the server can determine whether the target model needs to be adjusted in an optimized manner, that is, whether to generate second adjustment information, by monitoring the data, performance and time corresponding to the target model. The process of determining whether to generate the second adjustment information includes the following steps:

[0073] S40: Determine a performance characteristic value for characterizing performance fluctuations of the target model.

[0074] S41. Determine data feature values ​​for characterizing data fluctuations corresponding to the target model.

[0075] S42: Determine whether a preset time period is met.

[0076] S43. The above steps S40, S41 and S42 can be monitored simultaneously to determine whether there is at least one monitoring result that triggers the second adjustment condition.

[0077] S44: Generate second adjustment information when at least one second adjustment condition is triggered.

[0078] S45: When the second adjustment condition is not triggered at all, the second adjustment information is not generated.

[0079] Step S400: Determine target adjustment information according to the first adjustment information, so as to adjust the target model according to the target adjustment information.

[0080] Specifically, after the first adjustment information is generated, the server may determine the target adjustment information according to the first adjustment information, so as to adjust the target model through the target adjustment information. Further, after the second adjustment information is generated, the server determines the target adjustment information according to the first adjustment information and the second adjustment information, so as to adjust the target model through the target adjustment information. Optionally, the generation rule of the target adjustment information is that when the first adjustment information is generated and the second adjustment information is not generated, the first adjustment information is determined to be the target adjustment information. When both the first adjustment information and the second adjustment information are generated, the second adjustment information is determined to be the target adjustment information. When the second adjustment information is generated and the first adjustment information is not generated, the second adjustment information is determined to be the target adjustment information.

[0081] In an optional implementation of the embodiment of the present invention, the server determines that the first adjustment information is target adjustment information, so as to adjust the target model through the target adjustment information. Since the first adjustment information is an update adjustment, the server determines that the standby model corresponding to the standby model identifier included in the target adjustment information is the target standby model, and updates the target standby model to a new target model.

[0082] Furthermore, before performing the update adjustment, the server may also send an update request including the standby model identifier to the relevant terminal device, so that the operation and maintenance personnel using the terminal device can determine whether to update the model. When receiving confirmation information returned based on the update request, the target standby model is updated to the target model. Optionally, when receiving objection information returned based on the update request or not receiving confirmation information, the model update is not performed.

[0083] In another optional implementation of the embodiment of the present invention, the server determines the second adjustment information as the target adjustment information, so as to adjust the target model through the target adjustment information. The optimization updating method is to determine the adjustment sample set through the server, and the adjustment sample set includes at least one adjustment sample and a corresponding adjustment output. The target model is trained offline according to the adjustment sample set, and the target model is updated after the training is completed and the preset offline evaluation conditions are met. The adjustment sample set can be a periodically stored online sample set, or an offline sample set determined by an offline simulation method.

[0084] Figure 5 FIG. 1 is a flow chart of a process for determining target adjustment information according to an embodiment of the present invention. Figure 5 As shown, the process of the server determining the target adjustment information includes the following steps:

[0085] S50: Determine whether first adjustment information is generated.

[0086] S51: Determine whether second adjustment information is generated.

[0087] S52: When only the first adjustment information is generated, determine the first adjustment information as the target adjustment information.

[0088] S53: When only the second adjustment information is generated, or when both the first adjustment information and the second adjustment information are generated, determine that the second adjustment information is the target adjustment information.

[0089] The data processing method of the embodiment of the present invention monitors the model performance by determining the offline standby model corresponding to the target model, and monitors whether the target model needs to be adjusted by replacement when the performance of the standby model is better than the target model, and monitors whether the target model needs to be adjusted by optimization by determining the performance fluctuation and data fluctuation of the target model. The target model is automatically adjusted by optimization or replacement according to the monitoring results, so as to improve the performance of the online model and realize intelligent model monitoring, maintenance and management.

[0090] Figure 6 FIG. 1 is a schematic diagram of a data processing device according to an embodiment of the present invention. Figure 6 As shown, the model determination module 60, the model monitoring module 61, the information generation module 62 and the model adjustment module 63.

[0091] Specifically, the model determination module 60 is used to determine the target model and at least one corresponding backup model. The model monitoring module 61 is used to determine the performance difference value between each of the backup models and the target model, and the performance difference value is used to characterize the performance difference between the corresponding backup model and the target model. The information generation module 62 is used to generate first adjustment information including a backup model identifier in response to the performance difference satisfying the first adjustment condition, which is used to adjust the target model in an updated manner. The model adjustment module 63 is used to determine the target adjustment information according to the first adjustment information, so as to adjust the target model through the target adjustment information.

[0092] The corresponding offline standby model monitors the model performance, and monitors whether the target model needs to be adjusted by replacement when the performance of the standby model is better than the target model, and monitors whether the target model needs to be adjusted by optimization by determining the performance fluctuation and data fluctuation of the target model. And according to the monitoring results, the target model is automatically selected by optimization or replacement to improve the performance of the online model, realizing intelligent model monitoring, maintenance and management.

[0093] Figure 7 FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 7 As shown, Figure 7The electronic device shown is a general address query device, which includes a general computer hardware structure, which at least includes a processor 70 and a memory 71. The processor 70 and the memory 71 are connected via a bus 72. The memory 71 is suitable for storing instructions or programs executable by the processor 70. The processor 70 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 70 executes the instructions stored in the memory 71, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 72 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 73 and the display device and the input / output (I / O) device 74. The input / output (I / O) device 74 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 74 is connected to the system via an input / output (I / O) controller 75.

[0094] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0096] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0097] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0098] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0099] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Determine any one model from among multiple online models as a target model, where the target model is used to predict the processing time of an online ride-hailing order, predict the number of online ride-hailing orders within a period of time, predict a path from among multiple paths corresponding to an online ride-hailing order, or determine the probability of a negative review for an online ride-hailing order; Determine at least one backup model corresponding to the target model, the backup model playing the same role as the corresponding target model; Determine a performance difference value between each of the backup models and the target model, wherein the performance difference value is used to characterize a performance difference between the corresponding backup model and the target model; In response to the performance difference value satisfying a first adjustment condition, generating first adjustment information including a backup model identifier for adjusting the target model in an updated manner; Target adjustment information is determined according to the first adjustment information, so as to adjust the target model by using the target adjustment information.

2. The method according to claim 1, characterized in that Determining the performance difference between each of the backup models and the target model includes: Determine the first offline performance and the second offline performance corresponding to the target model and each of the backup models respectively according to a preset offline sample set, wherein the offline sample set includes at least one offline sample used as a model input and a corresponding offline output; A corresponding performance difference value is determined according to a difference between the first offline performance and each of the second offline performances.

3. The method according to claim 1, characterized in that In response to the performance difference satisfying the first adjustment condition, generating first adjustment information includes: Determining whether the performance difference value corresponding to each of the backup models is less than a preset first threshold; In response to the existence of a performance difference value less than a first threshold, first adjustment information including a backup model identifier corresponding to the performance difference value is generated.

4. The method according to claim 3, characterized in that In response to the performance difference value satisfying the first adjustment condition, generating the first adjustment information further includes: In response to the presence of a plurality of performance difference values ​​that are smaller than a first threshold, comparing the magnitudes of the performance difference values; First adjustment information including an identification of a backup model having a minimum performance difference value is determined.

5. The method according to claim 1, characterized in that: The method further comprises: In response to a second adjustment condition being triggered, second adjustment information for adjusting the target model in an optimized manner is generated.

6. The method according to claim 5, characterized in that The determining of the target adjustment information according to the first adjustment information is specifically: Target adjustment information is determined according to at least one of the first adjustment information and the second adjustment information, so as to adjust the target model by using the target adjustment information.

7. The method according to claim 5, characterized in that In response to the second adjustment condition being triggered, generating second adjustment information for adjusting the target model in an optimized manner comprises: Determining a performance characteristic value of the target model; In response to the performance characteristic value being greater than a second threshold, second adjustment information for adjusting the target model in an optimized manner is generated.

8. The method according to claim 7, characterized in that Determining the performance characteristic value of the target model includes: Determine an online sample set and historical performance parameters corresponding to the target model, wherein the online sample set includes at least one online sample for inputting the target model and a corresponding real result; Determine the current performance parameters of the target model according to the actual results corresponding to each of the online samples and the predicted results obtained by inputting the target model; A corresponding performance characteristic value is determined according to the historical performance parameter and the current performance parameter.

9. The method according to claim 5, characterized in that In response to the second adjustment condition being triggered, generating second adjustment information for adjusting the target model in an optimized manner comprises: Determining data feature values ​​corresponding to the target model; In response to the data feature value being greater than a third threshold, second adjustment information for adjusting the target model in an optimized manner is generated.

10. The method according to claim 9, characterized in that Determining the data features corresponding to the target model includes: Determining historical data and current data corresponding to the target model; A data characteristic value is determined according to the historical data and the current data, and the data characteristic value is used to characterize the difference between the historical data and the current data.

11. The method according to claim 5, characterized in that In response to the second adjustment condition being triggered, generating the second adjustment information for adjusting the target model in an optimized manner specifically comprises: Second adjustment information for adjusting the target model in an optimized manner is periodically generated according to a preset time period.

12. The method according to claim 6, characterized in that The determining target adjustment information according to at least one of the first adjustment information and the second adjustment information to adjust the target model by the target adjustment information comprises: In response to the second adjustment information not being determined, determining the first adjustment information as target adjustment information; In response to the second adjustment information being determined, determining the second adjustment information as target adjustment information; The target model is adjusted according to the target adjustment information.

13. The method according to claim 1, characterized in that The adjusting the target model by using the target adjustment information comprises: In response to the target adjustment information being the first adjustment information, determining a target backup model based on the backup model identifier in the target adjustment information; The target standby model is updated to the target model.

14. The method according to claim 13, characterized in that The updating of the target standby model to the target model comprises: Sending an update request including the standby model identifier; In response to receiving confirmation information returned based on the update request, the target standby model is updated to the target model.

15. The method according to claim 12, characterized in that The adjusting the target model by using the target adjustment information comprises: In response to the target adjustment information being the second adjustment information, determining an adjustment sample set, the adjustment sample set including at least one adjustment sample and a corresponding adjustment output; The target model is offline trained and updated according to the adjustment sample set.

16. A data processing device, characterized in that: The device comprises: A model determination module is used to determine any one model from multiple online models as a target model, wherein the target model is used to predict the processing time of an online car-hailing order, predict the number of online car-hailing orders within a period of time, predict a path from multiple paths corresponding to an online car-hailing order, or determine the probability of a bad review for an online car-hailing order, and determine at least one backup model corresponding to the target model, wherein the backup model plays the same role as the corresponding target model; A model monitoring module, used to determine a performance difference value between each of the backup models and the target model, wherein the performance difference value is used to characterize a performance difference between the corresponding backup model and the target model; an information generating module, configured to generate first adjustment information including a spare model identifier in response to the performance difference satisfying a first adjustment condition, for adjusting the target model in an updating manner; The model adjustment module is used to determine target adjustment information according to the first adjustment information, so as to adjust the target model according to the target adjustment information.

17. A computer-readable storage medium for storing computer program instructions, characterized in that: The computer program instructions, when executed by a processor, implement the method according to any one of claims 1 to 15.

18. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-15.

Images