Model parameter adjustment method and device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, and discloses a model parameter adjustment method and device, equipment and a medium, and the method comprises the steps: receiving a model parameter adjustment request initiated by a user in a visual interface, the model parameter adjustment request carrying the position of an initial model to be subjected to parameter adjustment and training parameter configuration; converting the model parameter adjustment request into a corresponding Spark parameter adjustment operation through Live service, and sending the Spark parameter adjustment operation to a Spark cluster; the Spark cluster obtains an initial model from the Hadoop distributed file system according to the Spark parameter adjustment operation, and performs distributed calculation on the initial model on different nodes of the cluster based on training parameter configuration until an optimal parameter combination is obtained; and storing the target model with the optimal parameter combination in a Hadoop distributed file system, and returning the target model to a user when a model downloading request is received. The parameter adjustment request of the user interface is converted into the Spark operation through the Live service, and the model parameter adjustment requirement under the conditions of massive data input and business adjustment is met by utilizing the distributed computing power of the Spark cluster.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a model parameter adjustment method, device, equipment and medium. Background Art

[0002] Offline prediction usually refers to the process of inputting a certain amount of existing data through batch processing tasks, and finally obtaining a training model through intensive calculations. In the field of finance and insurance, companies can use the model obtained by adjusting the offline model parameters to predict future product sales, number of users, etc. The offline prediction process includes a large number of calculations, repeated parameter adjustments, model result evaluation and other processes. The prediction parameter adjustment process is especially the stage where model trainers need to devote a lot of energy to repeatedly perform, and the changing data and the demand for accurate models further increase the workload of trainers.

[0003] The traditional parameter adjustment process is usually inefficient due to problems such as large amount of calculation, single-point calculation, and cumbersome system interaction. In addition, in the process of adjusting the use of computing resources, it may be necessary to learn additional server commands, etc., which makes model trainers invest a high time cost. At the same time, in the process of business development of production systems, as the types of business gradually diversify, it is necessary to frequently adjust the model according to the business situation, and the problem that follows is the increase in the workload of model parameter adjustment. Therefore, how to quickly adjust offline model parameters under the demand of large amounts of data input and frequent adjustment of model parameters according to business changes is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a model parameter adjustment method, device, equipment and medium that can be applied to financial technology or other related fields. Its main purpose is to achieve efficient offline model parameter adjustment to meet the model parameter adjustment needs under large amounts of data input and business adjustment conditions.

[0005] The technical solution of the present invention is as follows:

[0006] A first aspect of the present invention provides a model parameter adjustment method, comprising:

[0007] Receive a model parameter adjustment request initiated by a user in a visualization interface, wherein the model parameter adjustment request carries the location and training parameter configuration of an initial model to be adjusted;

[0008] The model parameter adjustment request is converted into a corresponding Spark parameter adjustment job through the Livy service and sent to the Spark cluster;

[0009] The Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until an optimal parameter combination is obtained;

[0010] The target model with the optimal parameter combination is stored in the Hadoop distributed file system, and the target model is returned to the user when a model download request is received.

[0011] A second aspect of the present invention provides a model parameter adjustment device, comprising:

[0012] A request receiving module, used to receive a model parameter adjustment request initiated by a user in a visualization interface, wherein the model parameter adjustment request carries the location and training parameter configuration of the initial model to be adjusted;

[0013] A request forwarding module is used to convert the model parameter adjustment request into a corresponding Spark parameter adjustment job through the Livy service and send it to the Spark cluster;

[0014] A distributed execution module, used to obtain the initial model from the Hadoop distributed file system through the Spark cluster according to the Spark parameter adjustment job, and perform distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until an optimal parameter combination is obtained;

[0015] The storage acquisition module is used to store the target model with the optimal parameter combination in the Hadoop distributed file system and return the target model to the user when receiving a model download request.

[0016] A third aspect of the present invention provides a computer device, comprising at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned model parameter adjustment method.

[0019] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned model parameter adjustment method.

[0020] Beneficial effects: The present invention discloses a model parameter adjustment method, device, equipment and medium. Compared with the prior art, the embodiment of the present invention receives a model parameter adjustment request initiated by a user in a visual interface, and the model parameter adjustment request carries the location and training parameter configuration of the initial model to be adjusted; the model parameter adjustment request is converted into a corresponding Spark parameter adjustment job through the Livy service and sent to the Spark cluster; the Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until the optimal parameter combination is obtained; the target model with the optimal parameter combination is stored in the Hadoop distributed file system, and the target model is returned to the user when a model download request is received. The parameter adjustment request of the user interface is converted into a Spark job through the Livy service, and the distributed computing capability of the Spark cluster is used to efficiently process a large amount of data input to quickly find the optimal parameter combination to complete the model parameter adjustment, and the adjusted model is stored in the Hadoop file system, which is convenient for users to download and use based on business needs, meeting the model parameter adjustment needs under the conditions of a large amount of data input and business adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of an application environment of the model parameter adjustment method provided by an embodiment of the present invention;

[0023] Figure 2 A flow chart of a model parameter adjustment method provided by an embodiment of the present invention;

[0024] Figure 3 A flow chart of step S202 in the model parameter adjustment method provided in an embodiment of the present invention;

[0025] Figure 4 A flow chart of step S203 in the model parameter adjustment method provided in an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the functional modules of a model parameter adjustment device provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0029] The model parameter adjustment method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment of FIG. 1 , it includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0031] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0032] The server 105 may be a server that provides various services, such as a background server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The background server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server 105 may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server 105 may also be a server for a distributed system, or a server combined with a blockchain.

[0033] It should be noted that the model parameter adjustment method provided in the embodiment of the present application can generally be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the model parameter adjustment device provided in the embodiment of the present invention can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103. Alternatively, the model parameter adjustment method provided in the embodiment of the present invention can also generally be executed by the server 105. Accordingly, the model parameter adjustment device provided in the embodiment of the present invention can generally be set in the server 105.

[0034] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0035] like Figure 2 As shown, the model parameter adjustment method provided by the embodiment of the present invention specifically includes the following steps:

[0036] S201. Receive a model parameter adjustment request initiated by a user in a visualization interface, wherein the model parameter adjustment request carries the location and training parameter configuration of an initial model to be adjusted.

[0037] In this embodiment, Web service is used as the interaction core to encapsulate all interactions between users and the parameter adjustment system, and a graphical visualization interface is provided to receive requests related to various functional modules issued by users. Through intuitive operations on the visualization interface, users can realize functions such as model uploading, parameter configuration, submission of model parameter adjustment requests, and viewing and downloading of final parameter adjustment results.

[0038] When a user initiates a model parameter adjustment request through the visual interface, the parameter adjustment system receives the model parameter adjustment request, which carries the location of the initial model to be adjusted and the training parameter configuration, where the initial model refers to a machine learning model, such as a logistic regression model, a decision tree model, an integrated model (such as random forest, XGBoost, LightGBM, CatBoost, etc.), etc. The training parameter configuration may include the training target (i.e., the parameters to be adjusted in the model) and the training parameters (such as the number of iterations, learning rate, optimizer, etc.). When initiating a parameter adjustment request for the same initial model, the user can specify a unique set of training parameters for training to quickly obtain the parameter adjustment results, or can select multiple sets of different training parameters for training to compare the model effects under different training parameter configurations. For example, in the financial risk control model training scenario, the user can upload an initial logistic regression model and provide a set of training parameter configurations, such as specifying the parameters to be adjusted and setting the number of iterations and learning rate during training.

[0039] Specifically, common prediction models, such as logistic regression models, decision tree models, and integrated models (such as random forests, XGBoost, LightGBM, CatBoost, etc.) can be set in advance in the Hadoop distributed file system. Users can directly select the corresponding initial model to be adjusted from the built-in prediction model through the visual interface according to business needs, and then obtain the address of the initial model in the Hadoop distributed file system; or, when the built-in model cannot meet the current business needs, the user can also upload a new custom model to the Hadoop distributed file system through the visual interface, and obtain the address of the custom model as the address of the initial model. Similarly, the training parameter configuration can be input by the user when initiating the model parameter adjustment request this time, or multiple groups of commonly used parameter configuration combinations can be set in advance in the system, so that the user can directly select the training parameter configuration required this time from the built-in parameter configuration combination, and this embodiment does not limit this. That is, in this embodiment, the user can flexibly select or input the initial model to be adjusted and the training parameter configuration according to different business needs and data characteristics, so as to adapt to the needs of model training for diverse business types.

[0040] S202. Convert the model parameter adjustment request into a corresponding Spark parameter adjustment job through the Livy service and send it to the Spark cluster.

[0041] In this embodiment, after receiving the model parameter adjustment request submitted by the user, the model parameter adjustment request is converted into a Spark parameter adjustment job through the Livy service and submitted to the Spark cluster. For example, in the training of the financial risk control model, the user hopes to optimize the regularization parameter of the logistic regression model. The Livy service converts this request into a Spark job, encapsulates the parameters and data reading path of the logistic regression model and other information into a job format that Spark can understand, and sends it to the Spark cluster. That is, the Livy service acts as an intermediate bridge, connecting the Web service and the Spark service, providing a RESTAPI interface for the Spark service, allowing users to submit jobs to the Spark cluster. The introduction of the Livy service gets rid of the Spark task submission form in the form of command line scripts, making the submission and operation monitoring of parameter adjustment tasks more portable. As an intermediate bridge, the service avoids the original way of submitting tasks through scripts, but presents the task interaction with Spark in the form of a REST API, which enables the Spark task operation to be realized through the Web service request Livy service, and also provides multi-user, security and fault-tolerant support for enterprise applications. Users do not need to learn the relevant scripting language for task operations, but can complete the configuration and submission of parameter adjustment task operation parameters through the page anytime and anywhere.

[0042] S203, the Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until an optimal parameter combination is obtained.

[0043] In this embodiment, the distributed big data computing framework Spark service is the training core, responsible for the training calculation of the model, and can be controlled by the Livy service. After receiving the Spark parameter adjustment job sent by the Livy service, the Spark cluster obtains the initial model and data from the Hadoop distributed file system based on the address information carried in the parameter adjustment job, and then performs distributed calculations on the initial model based on the training parameter configuration on different nodes. For example, in the training of financial risk control models, the Spark cluster can adjust the parameters of the initial model by processing a large amount of transaction data in parallel through multiple nodes when performing the parameter adjustment job, and find the optimal parameter combination based on the calculation results of all nodes through cross-validation or other methods, thereby optimizing the prediction ability of the model. Through distributed computing, the Spark cluster can effectively process large-scale data sets, accelerate the model training and parameter adjustment process, and ensure the scalability and flexibility of the model. Finding the optimal parameter combination through training and parameter adjustment ensures that the model performs best on the training data and has good generalization ability.

[0044] S204: Store the target model with the optimal parameter combination in the Hadoop distributed file system, and return the target model to the user when receiving a model download request.

[0045] In this embodiment, after the optimal parameter combination is calculated, the target model with the optimal parameter combination can be obtained, and the target model is stored in the Hadoop distributed file system. For example, the trained logistic regression model and its parameters are saved on HDFS, and the model is stored on HDFS. The high fault tolerance and high throughput of HDFS are used to effectively avoid data loss, and the needs of large-scale computing are fully adapted to ensure the accessibility and persistence of the model, while facilitating the version control of the model and subsequent model evaluation. When the user needs to use the target model, a model download request can be initiated through the visual interface, the target model can be retrieved from the Hadoop distributed file system, and the file of the target model can be transferred to the user, so that it can be used for further analysis or deployed in a production environment, providing users with convenient access to the model, so that users can conveniently and efficiently apply the trained model to subsequent business processes, such as risk assessment, credit approval, etc.

[0046] In the above embodiment, the present invention discloses a model parameter adjustment method, which receives a model parameter adjustment request initiated by a user in a visual interface, wherein the model parameter adjustment request carries the location of the initial model to be adjusted and the training parameter configuration; the model parameter adjustment request is converted into a corresponding Spark parameter adjustment job through the Livy service and sent to the Spark cluster; the Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until the optimal parameter combination is obtained; the target model with the optimal parameter combination is stored in the Hadoop distributed file system, and the target model is returned to the user when a model download request is received. The parameter adjustment request of the user interface is converted into a Spark job through the Livy service, and the distributed computing capability of the Spark cluster is utilized to efficiently process a large amount of data input to quickly find the optimal parameter combination to complete the model parameter adjustment, and the adjusted model is stored in the Hadoop file system, which is convenient for users to download and use based on business needs, and meets the model parameter adjustment needs under the conditions of a large amount of data input and business adjustment.

[0047] In one embodiment, before step S201, the method further includes:

[0048] Receiving a model upload operation and a parameter configuration operation input by a user on the visualization interface;

[0049] Acquire a corresponding model file according to the model upload operation, and acquire a corresponding parameter configuration combination according to the parameter configuration;

[0050] The currently acquired model file is stored in the Hadoop distributed file system, and the parameter configuration combination is stored in a specified location of the database.

[0051] In this embodiment, the user can upload one or more model files and parameter configurations in advance through the visualization operation of the graphical interface for subsequent flexible calls. For example, the user can click the "Upload Model" button on the visualization interface, select a model file (for example, a Python pickle file in .pkl format), and set the parameter configuration through the drop-down menu or input box, such as selecting the regularization strength as 0.1 and the number of iterations as 100. Allowing users to interact with the system through a friendly interface without directly handling the underlying complex operations, making the model training and parameter adjustment process more intuitive and easy to use.

[0052] According to the user's upload operation, the model file selected by the user is automatically identified and obtained. And according to the user's parameter configuration operation, the parameter configuration combination selected or input by the user is obtained, and the model file uploaded by the user is stored in the specified path in the Hadoop distributed file system (for example, / hdfs / models / credit_scoring_model.pkl). At the same time, the parameter configuration combination is stored in the specified table of the database and associated with the model file (for example, stored in the model_params table of the MySQL database, and the record ID is 1). That is, in this embodiment, the Hadoop distributed file system (HDFS) is used as the storage core, and the model file, data set, training results, etc. are stored therein, and the upload and download interaction of the model file, data set, training results, etc. can be realized through Web services; and the MySql database is used as the configuration storage core, and the model configuration parameters and other configurations required for the Web are stored therein, and the interaction of configuration data such as the model configuration parameters can be realized through Web services. By storing the model file in the Hadoop distributed file system, the high availability and scalability of the model file can be ensured, and the subsequent model management and version control can be facilitated. By storing the parameter configuration in the database, the parameter configuration of different models can be easily tracked and managed, and the model reproduction and parameter adjustment process can be supported.

[0053] In one embodiment, Figure 3 As shown, step S202 includes:

[0054] S301, receiving the model parameter adjustment request through the Livy service, and creating a corresponding session for the model parameter adjustment request, where the types of the session include a batch session and an interactive session;

[0055] S302, parsing and obtaining the position and training parameter configuration of the initial model to be adjusted in the model parameter adjustment request, and encapsulating the position and training parameter configuration of the initial model to be adjusted into a corresponding Spark parameter adjustment job;

[0056] S303. Send the Spark parameter adjustment job to the Spark cluster through the session, and monitor the execution status of the Spark parameter adjustment job.

[0057] In this embodiment, when a user initiates a parameter adjustment task request through the system web service, Livy, as a REST service, receives these requests through the API endpoint it provides and creates a session based on the user's request. This session can be an interactive session, which allows interactive execution of code snippets, or a batch session, which allows submission of complete model parameter adjustment jobs. The Livy service provides a RESTful interface that allows users to interact with the Spark cluster through a web interface without having to directly handle complex cluster management tasks, which simplifies user operations and makes the model parameter adjustment process more flexible and responsive.

[0058] The Livy service parses the user's parameter adjustment request and obtains the location of the initial model to be adjusted (for example, the path stored on HDFS) and the training parameter configuration. Then, Livy encapsulates this information into a Spark parameter adjustment job, which contains all the information required for model training, such as data sources, algorithm parameters, resource requirements, etc. This allows the user's parameter adjustment request to be converted into a job format that the Spark cluster can understand and execute, thereby ensuring that the user's parameter adjustment request can be effectively delivered to the Spark cluster for reliable parameter adjustment calculation. The Livy service sends the encapsulated Spark parameter adjustment job to the Spark cluster through the previously created session. The Spark cluster starts to execute the job, and during the execution process, the Livy service monitors the execution status of the job in real time, such as job progress, resource usage, whether errors occur, etc., so that users can understand the execution status of the model parameter adjustment job in real time, discover and handle possible problems in a timely manner, ensure the smooth progress of the model parameter adjustment job, accelerate the model training process, and improve the reliability of parameter adjustment.

[0059] In one embodiment, when multiple model parameter adjustment requests are received within the same time period, after step S302, the method further includes:

[0060] Adding multiple Spark parameter adjustment jobs converted from the multiple model parameter adjustment requests to a pre-configured job queue;

[0061] Configure a corresponding priority for each Spark parameter adjustment job in the job queue according to the preset priority strategy.

[0062] In this embodiment, when multiple model parameter adjustment requests are received in the same time period, the Livy service converts these requests into multiple Spark parameter adjustment jobs. These jobs are added to a pre-configured job queue. For example, two queues can be configured, and each queue can be set with different weights and capacity limits to control the resource allocation ratio. When the Spark parameter adjustment job is added to the job queue, it can be automatically allocated based on factors such as the number of jobs and remaining resources in each job queue, and the user can also specify the queue to which it belongs when submitting the parameter adjustment request. At this time, the parameter adjustment job is assigned to the job queue specified by the user. Through job queue management, the resource allocation ratio of different job queues can be effectively controlled to ensure that high-priority model parameter adjustment jobs can obtain sufficient computing resources and complete training quickly.

[0063] On the basis of configuring the job queue, the Livy service also configures the corresponding priority for each Spark parameter adjustment job in the job queue according to the preset priority strategy. For example, for the market risk assessment model that requires a quick response, its parameter adjustment job is given a higher priority (such as a priority value set to 1), while for the conventional model optimization job, it may be given a lower priority (such as a priority value set to 5). By configuring the priority of each Spark parameter adjustment job, the order in which each job is submitted to the Spark cluster can be controlled, thereby determining the execution order of the job, so that the job execution can be scheduled according to the business importance and urgency of the job, ensuring that key jobs can obtain resources first and complete quickly, which not only improves the response speed of key businesses, but also optimizes the overall job execution efficiency.

[0064] In one embodiment, sending the Spark parameter adjustment job to the Spark cluster through the session specifically includes:

[0065] According to the priority of each Spark parameter adjustment job in the job queue, the jobs are sent to the Spark cluster one by one in sequence through the corresponding session; or,

[0066] According to the priority of each Spark parameter adjustment job in the job queue, several Spark parameter adjustment jobs are submitted to the Spark cluster in parallel through the corresponding session.

[0067] In this embodiment, after obtaining multiple Spark parameter adjustment jobs and configuring their priorities, when sending the Spark parameter adjustment jobs to the Spark cluster through the session, the corresponding sending method can be flexibly selected according to different needs. One of them is to send them to the Spark cluster one by one in sequence through the corresponding session according to the priority of each Spark parameter adjustment job in the job queue. For example, the job queue includes two Spark parameter adjustment jobs, job A and job B. Job A involves an urgent credit scoring model update, so it is marked as high priority and needs to be executed immediately; job B is a regular priority and can be executed later. The Livy service first sends job A to the Spark cluster through a batch session. Once job A is completed, the Livy service sends job B to the Spark cluster through the same or another batch session. The sequential execution method ensures that high-priority jobs can start execution immediately without being delayed by waiting for low-priority jobs to complete. In financial risk control scenarios that require rapid response, it can help financial institutions quickly adapt to market changes and reduce potential risks.

[0068] The other method is to submit several Spark parameter adjustment jobs to the Spark cluster in parallel through the corresponding session according to the priority of each Spark parameter adjustment job in the job queue. For example, the job queue includes four Spark parameter adjustment jobs, and the priorities of job A, job B, job C, and job D decrease in sequence. According to the cluster resources and configuration, the Livy service can send two Spark parameter adjustment jobs to the Spark cluster for execution in parallel at one time. At this time, the Livy service submits jobs A and B to the Spark cluster for execution at the same time through two different sessions. After jobs A and B are completed, jobs C and D are submitted to the Spark cluster for execution at the same time. While ensuring that high-priority jobs can obtain resources first and complete quickly, the job execution efficiency is also maximized.

[0069] In one embodiment, Figure 4 As shown, step S203 includes:

[0070] S401, acquiring the initial model and a training set for training the initial model from the Hadoop distributed file system according to the location of the initial model to be adjusted in the Spark parameter adjustment job;

[0071] S402, decomposing the Spark parameter adjustment job into multiple parameter adjustment subtasks and dividing the training set into multiple partition data;

[0072] S403, distributing the multiple parameter adjustment subtasks and the corresponding partition data to different nodes of the cluster, and executing the parameter adjustment subtask on the initial model using the corresponding partition data according to the training parameter configuration on each node to obtain the parameter adjustment result of each node;

[0073] S404: Aggregate the parameter adjustment results of each node and determine the optimal parameter combination based on the aggregation results.

[0074] In this embodiment, the Spark cluster obtains the initial model file (for example, a gradient boosting tree model) and the training data set from the Hadoop distributed file system according to the requirements of the parameter adjustment operation. This training data set includes data used for model parameter adjustment training, such as customer transaction records, credit scores, historical default information, etc. The initial model and training set are stored in different paths of HDFS respectively to ensure accurate acquisition of the model files and training data required for model training. In the Spark cluster, the Spark parameter adjustment operation is decomposed into multiple subtasks. For example, different learning rates and regularization parameter combinations can be tried through multiple subtasks. At the same time, the training set is also divided into multiple partitioned data, each partition is assigned to a different subtask. By dividing the training set into multiple partitions, the distributed computing power of the Spark cluster can be used to process a large amount of data at the same time, significantly improving data processing efficiency and model training speed.

[0075] After task decomposition and data partitioning, each parameter adjustment subtask and the corresponding partition data are assigned to different nodes of the cluster. On each node, the initial model is adjusted using the corresponding partition data according to the training parameter configuration (such as different learning rate and regularization parameter combinations). For example, node 1 may use a learning rate of 0.1 and an L2 regularization parameter of 0.01, node 2 may use a learning rate of 0.05 and an L2 regularization parameter of 0.001, and so on. The distributed parameter adjustment method allows multiple parameter combinations to be tested simultaneously, greatly speeding up the process of finding the optimal parameter combination. Each node works independently and can process different parameter combinations in parallel, improving the overall parameter adjustment efficiency.

[0076] After all nodes complete the parameter tuning subtask, the Spark cluster aggregates the parameter tuning results of each node. For example, the cluster may compare the model performance indicators (such as AUC, accuracy, and recall) of all nodes and determine that the combination of learning rate 0.05 and L2 regularization parameter 0.01 used on node 3 produces the best model performance. By aggregating and comparing the parameter tuning results of different parameter combinations, the Spark cluster is able to determine the optimal model parameter combination, ensuring that the final selected model parameters not only perform well on a single data partition, but also have the best performance on the overall data set, thereby improving the generalization ability and prediction accuracy of the model.

[0077] In one embodiment, after allocating the multiple parameter adjustment subtasks and the corresponding partition data to different nodes of the cluster, the method further includes:

[0078] Match the parameter adjustment subtasks and partition data of each node with the historical execution data, and estimate the resource requirements of each node based on the matching results;

[0079] Generate an initial resource allocation plan according to the resource demand of each node, and allocate corresponding initial resources to each node according to the initial resource allocation plan;

[0080] Monitor the resource usage rate and / or task progress of each node when executing the parameter adjustment subtask, and dynamically adjust the initial resources according to the resource usage rate and / or task progress of each node.

[0081] In this embodiment, after allocating the subtasks and partition data of each node, the parameter adjustment subtasks and partition data of each node are matched with the historical execution data, and the resource requirements of each node are estimated accordingly. For example, a complex risk control model parameter adjustment job is decomposed into multiple subtasks, each subtask is trained and verified for different parameter combinations of the model, and the amount of resources that may be required for each subtask when running on different nodes is estimated based on the characteristics of each subtask (such as data partitioning, pre-calculation amount) and historical execution data. If historical data shows that subtasks that process specific data sets usually require more memory, more memory resources are allocated to these subtasks. By estimating the resource requirements of each subtask, the most appropriate resources can be allocated to each subtask, ensuring that they can run efficiently without being limited by insufficient resources, while also avoiding waste of resources.

[0082] Based on the estimated results, an initial resource allocation plan is generated for each subtask of the parameter adjustment job. For example, the amount of resources to be allocated for each node is generated based on the current load and resource capacity of each node and the estimated resource demand. After the subtasks are assigned to different nodes, the appropriate amount of resources, such as CPU cores and memory, can be allocated to each node according to the amount of resources to be allocated, ensuring that each subtask can obtain the resources required for execution. In addition, the resource allocation takes into account the current state of the cluster to avoid overloaded nodes.

[0083] In addition, during the execution of the parameter adjustment subtask, the resource utilization and task progress of each node are continuously monitored. If it is found that the resource utilization of a node is much lower than that of other nodes, or a subtask completes its training phase ahead of schedule and releases resources, these resources can be dynamically reallocated to subtasks with higher resource utilization, or future resource requirements can be predicted and resources adjusted in advance. This allows computing resources to be flexibly allocated according to actual usage, improves resource utilization efficiency, reduces subtask delays caused by insufficient resources, and can also respond to sudden changes in resource demand, allowing model parameter adjustment to adapt to changes in market business more quickly.

[0084] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0085] Further references Figure 5 , as a response to the above Figure 2 The present invention provides an embodiment of a model parameter adjustment device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0086] like Figure 5 As shown, the model parameter adjustment device 50 described in this embodiment includes:

[0087] The request receiving module 501 is used to receive a model parameter adjustment request initiated by a user in a visualization interface, wherein the model parameter adjustment request carries the location and training parameter configuration of the initial model to be adjusted;

[0088] The request forwarding module 502 is used to convert the model parameter adjustment request into a corresponding Spark parameter adjustment job through the Livy service and send it to the Spark cluster;

[0089] A distributed execution module 503 is used to obtain the initial model from the Hadoop distributed file system through the Spark cluster according to the Spark parameter adjustment job, and perform distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until an optimal parameter combination is obtained;

[0090] The storage acquisition module 504 is used to store the target model with the optimal parameter combination in the Hadoop distributed file system, and return the target model to the user when receiving a model download request.

[0091] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which are more suitable for describing the model parameter adjustment execution process than programs. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0092] In one embodiment, the device 50 comprises:

[0093] An operation receiving module, used for receiving a model upload operation and a parameter configuration operation input by a user on the visual interface;

[0094] An acquisition module, used to acquire a corresponding model file according to the model upload operation, and acquire a corresponding parameter configuration combination according to the parameter configuration;

[0095] The storage module is used to store the currently acquired model file in the Hadoop distributed file system and store the parameter configuration combination in a specified location of the database.

[0096] In one embodiment, the request forwarding module 502 includes:

[0097] A creating unit, configured to receive the model parameter adjustment request through a Livy service, and create a corresponding session for the model parameter adjustment request, where the types of the session include a batch session and an interactive session;

[0098] A parsing and encapsulation unit, used for parsing and obtaining the position and training parameter configuration of the initial model to be adjusted in the model parameter adjustment request, and encapsulating the position and training parameter configuration of the initial model to be adjusted into a corresponding Spark parameter adjustment job;

[0099] A sending unit is used to send the Spark parameter adjustment job to the Spark cluster through the session and monitor the execution status of the Spark parameter adjustment job.

[0100] In one embodiment, when multiple model parameter adjustment requests are received in the same time period, the request forwarding module 502 further includes:

[0101] A job adding unit, used for adding a plurality of Spark parameter adjustment jobs converted from the plurality of model parameter adjustment requests to a pre-configured job queue;

[0102] The priority configuration unit is used to configure the corresponding priority for each Spark parameter adjustment job in the job queue according to a preset priority strategy.

[0103] In one embodiment, the sending unit is specifically configured to:

[0104] According to the priority of each Spark parameter adjustment job in the job queue, the jobs are sent to the Spark cluster one by one in sequence through the corresponding session; or,

[0105] According to the priority of each Spark parameter adjustment job in the job queue, several Spark parameter adjustment jobs are submitted to the Spark cluster in parallel through the corresponding session.

[0106] In one embodiment, the distributed execution module 503 includes:

[0107] An acquisition unit, used for acquiring the initial model and a training set for training the initial model from the Hadoop distributed file system according to the location of the initial model to be adjusted in the Spark parameter adjustment job;

[0108] A partitioning unit, used to decompose the Spark parameter adjustment job into multiple parameter adjustment subtasks and divide the training set into multiple partition data;

[0109] An allocation execution unit is used to allocate the multiple parameter adjustment subtasks and the corresponding partition data to different nodes of the cluster, and execute the parameter adjustment subtask on the initial model using the corresponding partition data according to the training parameter configuration on each node to obtain the parameter adjustment result of each node;

[0110] The aggregation unit is used to aggregate the parameter adjustment results of each node and determine the optimal parameter combination based on the aggregation results.

[0111] In one embodiment, the distributed execution module 503 further includes:

[0112] The resource estimation unit is used to match the parameter adjustment subtasks and partition data of each node with the historical execution data, and estimate the resource requirements of each node based on the matching results;

[0113] An initial allocation unit, used to generate an initial resource allocation plan according to the resource demand of each node, and allocate corresponding initial resources to each node according to the initial resource allocation plan;

[0114] A monitoring and adjustment unit is used to monitor the resource utilization rate and / or task progress of each node when executing the parameter adjustment subtask, and dynamically adjust the initial resources according to the resource utilization rate and / or task progress of each node.

[0115] In the above embodiment, the present invention discloses a model parameter adjustment device, which receives a model parameter adjustment request initiated by a user in a visual interface, wherein the model parameter adjustment request carries the location of the initial model to be adjusted and the training parameter configuration; the model parameter adjustment request is converted into a corresponding Spark parameter adjustment job through the Livy service and sent to the Spark cluster; the Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until the optimal parameter combination is obtained; the target model with the optimal parameter combination is stored in the Hadoop distributed file system, and the target model is returned to the user when a model download request is received. The parameter adjustment request of the user interface is converted into a Spark job through the Livy service, and the distributed computing capability of the Spark cluster is utilized to efficiently process a large amount of data input to quickly find the optimal parameter combination to complete the model parameter adjustment, and the adjusted model is stored in the Hadoop file system, which is convenient for users to download and use based on business needs, and meets the model parameter adjustment needs under the conditions of a large amount of data input and business adjustment.

[0116] Another embodiment of the present invention provides a computer device, such as Figure 6 As shown, the computer device 60 includes:

[0117] One or more processors 601 and memory 602, Figure 6 A processor 601 is used as an example for the description. The processor 601 and the memory 602 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0118] Processor 601 is used to complete various control logics of computer device 60, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component or any combination of these components. In addition, processor 601 can also be any traditional processor, microprocessor or state machine. Processor 601 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0119] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the model parameter adjustment method in the embodiment of the present invention. The processor 601 executes various functional applications and data processing of the computer device 60 by running the non-volatile software programs, instructions and units stored in the memory 602, that is, implements the model parameter adjustment method in the above method embodiment.

[0120] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device 60, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the computer device 60 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. One or more units are stored in the memory 602, and when executed by one or more processors 601, the steps of the model parameter adjustment method in any of the above-mentioned method embodiments are executed.

[0121] In the above embodiment, the present invention discloses a computer device, which receives a model parameter adjustment request initiated by a user in a visual interface, wherein the model parameter adjustment request carries the location of the initial model to be adjusted and the training parameter configuration; converts the model parameter adjustment request into a corresponding Spark parameter adjustment job through the Livy service and sends it to the Spark cluster; the Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until the optimal parameter combination is obtained; stores the target model with the optimal parameter combination in the Hadoop distributed file system, and returns the target model to the user when receiving the model download request. The parameter adjustment request of the user interface is converted into a Spark job through the Livy service, and the distributed computing capability of the Spark cluster is utilized to efficiently process a large amount of data input to quickly find the optimal parameter combination to complete the model parameter adjustment, and the adjusted model is stored in the Hadoop file system, which is convenient for users to download and use based on business needs, and meets the model parameter adjustment needs under the conditions of a large amount of data input and business adjustment.

[0122] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the model parameter adjustment method in any of the above method embodiments are executed.

[0123] In the above embodiment, the present invention discloses a non-volatile computer-readable storage medium, which receives a model parameter adjustment request initiated by a user in a visual interface, wherein the model parameter adjustment request carries the location of the initial model to be adjusted and the training parameter configuration; the model parameter adjustment request is converted into a corresponding Spark parameter adjustment job through the Livy service and sent to the Spark cluster; the Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until the optimal parameter combination is obtained; the target model with the optimal parameter combination is stored in the Hadoop distributed file system, and the target model is returned to the user when a model download request is received. The parameter adjustment request of the user interface is converted into a Spark job through the Livy service, and the distributed computing capability of the Spark cluster is utilized to efficiently process a large amount of data input to quickly find the optimal parameter combination to complete the model parameter adjustment, and the adjusted model is stored in the Hadoop file system, which is convenient for users to download and use based on business needs, and meets the model parameter adjustment needs under the conditions of a large amount of data input and business adjustment.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0125] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0126] In summary, the present invention discloses a model parameter adjustment method, device, equipment and medium, the method includes: receiving a model parameter adjustment request initiated by a user in a visual interface, the model parameter adjustment request carries the location of the initial model to be adjusted and the training parameter configuration; converting the model parameter adjustment request into a corresponding Spark parameter adjustment job through the Livy service and sending it to the Spark cluster; the Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until the optimal parameter combination is obtained; the target model with the optimal parameter combination is stored in the Hadoop distributed file system, and the target model is returned to the user when a model download request is received. The parameter adjustment request of the user interface is converted into a Spark job through the Livy service, and the distributed computing capability of the Spark cluster is utilized to efficiently process a large amount of data input to quickly find the optimal parameter combination to complete the model parameter adjustment, and the adjusted model is stored in the Hadoop file system, which is convenient for users to download and use based on business needs, and meets the model parameter adjustment needs under the conditions of a large amount of data input and business adjustment.

[0127] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0128] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of the present application, they are merely used for illustration and do not represent actual use. It should be understood that the application of the present invention is not limited to the above examples, and that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes shall fall within the scope of protection of the claims attached to the present invention.

Claims

1. A model parameter adjustment method, characterized in that: include: Receive a model parameter adjustment request initiated by a user in a visualization interface, wherein the model parameter adjustment request carries the location and training parameter configuration of an initial model to be adjusted; The model parameter adjustment request is converted into a corresponding Spark parameter adjustment job through the Livy service and sent to the Spark cluster; The Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until an optimal parameter combination is obtained; The target model with the optimal parameter combination is stored in the Hadoop distributed file system, and the target model is returned to the user when a model download request is received.

2. The model parameter adjustment method according to claim 1, characterized in that: Before receiving a model parameter adjustment request initiated by a user in a visualization interface and determining an initial model and training parameter configuration to be adjusted that are pre-stored in a Hadoop distributed file system according to the model parameter adjustment request, the method further includes: Receiving a model upload operation and a parameter configuration operation input by a user on the visualization interface; Acquire a corresponding model file according to the model upload operation, and acquire a corresponding parameter configuration combination according to the parameter configuration; The currently acquired model file is stored in the Hadoop distributed file system, and the parameter configuration combination is stored in a specified location of the database.

3. The model parameter adjustment method according to claim 1, characterized in that: The step of converting the model parameter adjustment request into a corresponding Spark parameter adjustment job through the Livy service and sending the job to the Spark cluster includes: Receive the model parameter adjustment request through the Livy service, and create a corresponding session for the model parameter adjustment request, where the session types include batch processing session and interactive session; Parse and obtain the position and training parameter configuration of the initial model to be adjusted in the model parameter adjustment request, and encapsulate the position and training parameter configuration of the initial model to be adjusted into a corresponding Spark parameter adjustment job; The Spark parameter adjustment job is sent to the Spark cluster through the session, and the execution status of the Spark parameter adjustment job is monitored.

4. The model parameter adjustment method according to claim 3, characterized in that: When multiple model parameter adjustment requests are received in the same time period, the method further includes: parsing to obtain the position and training parameter configuration of the initial model to be adjusted in the model parameter adjustment request, and encapsulating the position and training parameter configuration of the initial model to be adjusted into a corresponding Spark parameter adjustment job. Adding multiple Spark parameter adjustment jobs converted from the multiple model parameter adjustment requests to a pre-configured job queue; Configure a corresponding priority for each Spark parameter adjustment job in the job queue according to the preset priority strategy.

5. The model parameter adjustment method according to claim 4, characterized in that: The sending the Spark parameter adjustment job to the Spark cluster through the session specifically includes: According to the priority of each Spark parameter adjustment job in the job queue, the jobs are sent to the Spark cluster one by one in sequence through the corresponding session; or, According to the priority of each Spark parameter adjustment job in the job queue, several Spark parameter adjustment jobs are submitted to the Spark cluster in parallel through the corresponding session.

6. The model parameter adjustment method according to claim 1, characterized in that: The Spark cluster obtains the initial model from the Hadoop distributed file system according to the Spark parameter adjustment job, and performs distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until an optimal parameter combination is obtained, including: Obtaining the initial model and a training set for training the initial model from the Hadoop distributed file system according to the location of the initial model to be adjusted in the Spark parameter adjustment job; Decomposing the Spark parameter adjustment job into multiple parameter adjustment subtasks and dividing the training set into multiple partition data; Distribute the multiple parameter adjustment subtasks and corresponding partition data to different nodes of the cluster, and perform the parameter adjustment subtask on the initial model using the corresponding partition data according to the training parameter configuration on each node to obtain the parameter adjustment result of each node; Aggregate the parameter adjustment results of each node and determine the optimal parameter combination based on the aggregation results.

7. The model parameter adjustment method according to claim 6, characterized in that: After allocating the multiple parameter adjustment subtasks and the corresponding partition data to different nodes of the cluster, the method further includes: Match the parameter adjustment subtasks and partition data of each node with the historical execution data, and estimate the resource requirements of each node based on the matching results; Generate an initial resource allocation plan according to the resource demand of each node, and allocate corresponding initial resources to each node according to the initial resource allocation plan; Monitor the resource usage rate and / or task progress of each node when executing the parameter adjustment subtask, and dynamically adjust the initial resources according to the resource usage rate and / or task progress of each node.

8. A model parameter adjustment device, characterized in that: include: A request receiving module, used to receive a model parameter adjustment request initiated by a user in a visualization interface, wherein the model parameter adjustment request carries the location and training parameter configuration of the initial model to be adjusted; A request forwarding module is used to convert the model parameter adjustment request into a corresponding Spark parameter adjustment job through the Livy service and send it to the Spark cluster; A distributed execution module, used to obtain the initial model from the Hadoop distributed file system through the Spark cluster according to the Spark parameter adjustment job, and perform distributed computing on the initial model based on the training parameter configuration on different nodes of the cluster until an optimal parameter combination is obtained; The storage acquisition module is used to store the target model with the optimal parameter combination in the Hadoop distributed file system and return the target model to the user when receiving a model download request.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model parameter adjustment method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the model parameter adjustment method described in any one of claims 1-7.

Images