Model generation method and system and computer readable storage medium
By obtaining the model training data set and determining the standardized computing environment, the target model is debugged and generated, and the problem of poor maintainability in multi-model risk control scenarios is solved, and the standardization and efficient generation of the model are achieved.
Patent Information
- Application Number
- CN202510003734.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-02
- Publication Date
- 2025-05-06
AI Technical Summary
In risk control scenarios, when multiple models are used to analyze potential risks, the model is poor maintainability due to the lack of unified specifications for algorithms and software versions.
By obtaining the model training data set, the standardized computing environment for the target model to be generated is determined, and the target model is debugged and generated based on the standardized computing environment and target training data to achieve standardization and unification of the model.
It improves the maintainability of the model, unifies the algorithm and software version, so that the generated target model has a standard computing environment, which is convenient for maintenance, and automatically generates the target model, reducing manual intervention, and improving generation efficiency and reliability.
Smart Images
Figure CN119939249A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a model generation method, system and computer-readable storage medium. Background Art
[0002] Enterprises usually need to use models to analyze potential risks in risk control scenarios, and when analyzing potential risks, they need to use multiple models to analyze potential risks. Due to the lack of unified specifications for the algorithms and software versions used by multiple models for analyzing potential risks, the algorithms and software versions used by each model may be different, and different methods need to be used for maintenance, which reduces the maintainability of the model. Therefore, how to improve the maintainability of the model has become a technical problem that needs to be solved at present. Summary of the invention
[0003] In view of the above problems, the purpose of this application is to provide a model generation method, system and computer-readable storage medium to improve the maintainability of the model. The specific scheme is as follows:
[0004] In a first aspect, an embodiment of the present application provides a model generation method, the method comprising:
[0005] Get the model training dataset;
[0006] Determine a standardized computing environment for the target model to be generated;
[0007] Based on the standardized computing environment and target training data, the target model to be generated is debugged to generate a target model; the target training data is the data in the model training data set.
[0008] Optionally, obtaining a model training data set includes:
[0009] Creating a data table; the data table includes data to be imported;
[0010] Importing the data table into a pre-established data set;
[0011] The data in the data set is cleaned to obtain the model training data set.
[0012] Optionally, the cleaning of the data in the data set to obtain the model training data set includes:
[0013] Verify the data in the data set based on the custom rules to obtain a verification data set;
[0014] The data in the verification data set whose data similarity is higher than the similarity threshold are merged to obtain the model training data set.
[0015] Optionally, the method further includes:
[0016] Obtain the development image, working directory and mounting directory of the model to be entered;
[0017] Based on the development image, working directory and mounting directory of the model to be imported, the model to be imported is imported; the model to be imported includes the target model to be generated.
[0018] Optionally, determining a standardized computing environment for a target model to be generated includes:
[0019] Obtaining a custom computing environment selection result for the target model to be generated;
[0020] Based on the customized computing environment selection result, a standardized development environment and a standardized compilation environment are determined.
[0021] Optionally, debugging the target model to be generated based on the computing environment and the target training data includes:
[0022] Selecting the target model to be generated from the entered models;
[0023] In the standardized development environment and the standardized compilation environment, the target model to be generated is trained based on the target training data.
[0024] In a second aspect, an embodiment of the present application provides a model generation system, the system comprising:
[0025] Data management module, used to obtain model training data sets;
[0026] An online development module for determining a standardized computing environment for the target model to be generated;
[0027] The model training module is used to debug the target model to be generated based on the computing environment and target training data to generate the target model; the target training data is the data in the model training data set.
[0028] Optionally, the online development module is specifically used for:
[0029] Obtaining a custom computing environment selection result for the target model to be generated;
[0030] Based on the customized computing environment selection result, a standardized development environment and a standardized compilation environment are determined.
[0031] Optionally, the model training module is specifically used for:
[0032] Selecting the target model to be generated from the entered models;
[0033] In the standardized development environment and the standardized compilation environment, the target model to be generated is trained based on the target training data.
[0034] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any one of the above-described model generation methods.
[0035] Compared with the prior art, this application has the following beneficial effects:
[0036] By obtaining the model training data set, determining the standardized computing environment of the target model to be generated, debugging the target model to be generated based on the standardized computing environment and target training data, the target model is automatically generated based on the standardized computing environment, unifying the algorithm and software version, so that the generated target model has a standard computing environment, facilitating the maintenance of the target model and improving the maintainability of the model. In addition, by automatically generating the target model, manual intervention is reduced, and the generation efficiency and reliability of the target model are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0038] Figure 1 A schematic diagram of a flow chart of a model generation method provided in an embodiment of the present application;
[0039] Figure 2 A schematic diagram of the structure of the model generation system provided in an embodiment of the present application;
[0040] Figure 3 A schematic diagram of the structure of a data management module provided in an embodiment of the present application;
[0041] Figure 4 A schematic diagram of the structure of the online development module provided in the embodiment of the present application;
[0042] Figure 5 A schematic diagram of the structure of the model training module provided in the embodiment of the present application;
[0043] Figure 6 A schematic diagram of the structure of the distributed technology framework module provided in the embodiment of the present application;
[0044] Figure 7A schematic diagram of the structure of the cloud resource coordination module provided in the embodiment of the present application;
[0045] Figure 8 A schematic diagram of the structure of a model generation system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.
[0048] After analyzing the prior art, the inventor of this application found that in the prior art, since many models are applied in risk control scenarios and the algorithms and versions used in the models are different, the generated models lack unified specifications, which is not convenient for subsequent maintenance of the models and has poor maintainability. Therefore, in order to improve the maintainability of the model, this application provides a model generation method, system and computer-readable storage medium, as follows.
[0049] like Figure 1 As shown, the embodiment of the present application provides a model generation method, which is applied to a model generation system, and the method includes:
[0050] S101: Obtain a model training data set.
[0051] The model trainer creates a data task and creates a data set based on the selected data to obtain a model training data set. For example, the model trainer enters the Hive Structured Query Language (HiveSQL) used by the platform to configure the model training data set to be downloaded to the specified target physical machine, which can be a local machine or other server. During the execution of the training task, the model training data set downloaded to the specified target physical machine can be used by specifying the unique code (Identity Document, ID) of the data task created by the model trainer.
[0052] The model generation method in this application is applied to a model generation system, such as Figure 2 As shown in Figure 1, the model generation system includes a data management module, an online development module, a model training module, a distributed technology framework module, and a cloud native resource coordination module. Figure 3 As shown in FIG, the data management module is used to create data tables and data sets, perform feature verification, search for similar features, and optimize the Structured Query Language (SQL). Figure 4 As shown in the figure, the online development module is used to provide online notebooks, image repositories, image building, and image management functions. Figure 5 As shown in the figure, the model training module is used for model management, task management, and real-time monitoring. The distributed technology framework is used to provide software libraries. The cloud native resource coordination module is used to provide the hardware resources required for the model training process.
[0053] In an optional embodiment, obtaining a model training data set includes:
[0054] Create a data table; the data table includes the data to be imported;
[0055] Import data tables into pre-built datasets;
[0056] Clean the data in the dataset to obtain the model training dataset.
[0057] Specifically, creating a data table includes:
[0058] The data management module creates a data table based on the library name, table name, subject, and field. The data table includes the data to be imported. The data to be imported is the feature data generated in advance based on the sample data by the model trainer when training the model. For example, the data of a certain feature is generated in advance based on the HiveSQL statement or SQL statement, data source, and data source address.
[0059] It should be noted that when the data to be imported is generated, you can execute the query command according to the specified data source to copy the queried data to the local machine or the specified directory of other servers. Or, enter HiveSQL on the platform, and then download the data to be imported to the specified directory of the specified physical machine by configuring the location parameters.
[0060] If the data to be imported is obtained through SQL, it is necessary to optimize and check for abnormalities in SQL before obtaining the data to be imported to improve operating efficiency and fault tolerance. For example, check whether the SQL is legal and optimize SQL performance.
[0061] Specifically, importing a data table into a pre-established data set includes:
[0062] Import the generated data table into a pre-established data set. The import method can be set based on actual needs. For example, import the data table into a pre-established data set by uploading a file or using the HiveSQL query language, and archive it to the corresponding subject and field.
[0063] Specifically, the data in the data set is cleaned to obtain the model training data set including:
[0064] Verify the data in the data set based on the custom rules to obtain a verification data set;
[0065] The data in the verification data set whose data similarity is higher than the similarity threshold are merged to obtain the model training data set.
[0066] After the data table is imported into the data set, the data management module can be used to verify the content of the data in the data set based on the custom verification rules to obtain the verification data set. Similar features are searched in the verification data set to identify and locate duplicate or highly similar data in the data set, and duplicate or highly similar data are deleted or merged to delete, merge or otherwise process similar features, reduce the impact of redundant data on subsequent model training, and improve the quality of the model training data set.
[0067] For example, taking the example of a custom verification rule for checking the integrity and correctness of the data in the data set, this application verifies the integrity and correctness of the data in the data set after importing the data table into the data set to obtain a verification data set. Any two data in the verification data set are compared for similarity. If the similarity is greater than the similarity threshold, it means that the similarity between the two data compared this time is too high or the two data are duplicate data. The two data with similarity greater than the preset threshold are merged. Repeat the comparison step until the similarity comparison is completed between any two data in the data set. Thus, by importing, exporting, feature verification, similar feature search and SQL optimization of data, a full-cycle process for generating data to be imported is constructed to provide data support for the training and development of the target model.
[0068] S102: Determine a standardized computing environment for generating a target model.
[0069] The model generation system includes a variety of standardized computing environments. Model trainers can select different (notebooks) based on actual needs through the online development module, such as the open source interactive programming environment Jupyter, the cross-platform code editor VScode, etc., to adjust the program and parameters and perform visual operations of the program. In addition, since the model generation system includes a variety of standardized computing environments, model trainers can select the required computing environment from a variety of standardized computing environments as the standardized computing environment for the target model to be generated, thereby unifying the algorithms and software versions used by model trainers by adopting a standardized computing environment.
[0070] It should be noted that model trainers can customize the resources required for model training development and compilation based on the model type, such as memory, image files, central processing unit (CPU) and graphics processing unit (GPU). Image files are packaged files that contain applications and standardized computing environments. The online development module stores the available image files in the image warehouse in advance, and model training users can manage and build image files in the image warehouse, for example, customize the selection of image files and the parameters of image files.
[0071] In an optional embodiment, determining a standardized computing environment for generating a target model includes:
[0072] Obtaining a custom computing environment selection result for a target model to be generated;
[0073] Based on the custom computing environment selection results, determine the standardized development environment and standardized compilation environment.
[0074] The standardized computing environment includes a standardized development environment and a standardized compilation environment, such as tools such as Jupyter and VScode. After receiving the computing environment selection signal triggered by the model trainer, the computing environment selection result of the model trainer is used as the custom computing environment selection result for the target model to be generated, and the standardized development environment and the standardized compilation environment are extracted from the custom computing environment selection result. Thus, the standardized development environment and the standardized compilation environment are determined through the custom computing environment selection result, so that the standardized compilation environment can realize the elastic selection of memory size, number and model of CPU and GPU based on actual needs, ensuring that the model training process can be carried out stably, avoiding excessive waste of hardware resources, and enabling the standardized development environment to select different online notebooks based on actual needs.
[0075] S103: Based on the standardized computing environment and target training data, the target model to be generated is debugged to generate the target model; the target training data is the data in the model training data set.
[0076] Edit the parameters of the target model to be generated, and run the model once or periodically based on the standardized computing environment and target training data. Editing the parameters of the target model to be generated includes editing the scheduling machine, memory, CPU, GPU, and scheduled tasks of the target model to be generated.
[0077] In an optional embodiment, before generating the target model, the method further includes inputting the target model to be generated into the model generation system in advance, specifically:
[0078] Get the development image, working directory, and mount directory of the model to be imported;
[0079] The model to be imported is imported based on the development image, working directory and mounting directory of the model to be imported; the model to be imported includes the target model to be generated.
[0080] The development image is used to determine the software environment and set of dependent packages required for model development and training. The working directory is used to specify the file path and directory structure used in the model development and training process. The mount directory is used to define the data storage and access path required by the model in the containerized environment. The model training module in the model generation system enters the models to be entered according to the development image, working directory, mount directory and other parameters of each model to be entered, and provides editing functions to facilitate the entry of developed models into the model generation system to achieve model management of the entered models.
[0081] In an optional embodiment, debugging the target model to be generated based on the computing environment and the target training data includes:
[0082] Select the target model to be generated from the entered models;
[0083] In a standardized development environment and a standardized compilation environment, the target model to be generated is trained based on the target training data.
[0084] The target model to be generated is the model that has been entered into the model generation system in advance. The model trainer selects the model from the models that have been entered into the model generation system as the target model to be generated. The model trainer performs task management in a standardized development environment and a standardized compilation environment. The task management functions in the model training module include publishing tasks and debugging tasks. Publishing tasks include creating publishing tasks, modifying publishing tasks, deploying publishing tasks, and deleting publishing tasks, etc., to provide online services, view service status, and log monitoring functions. Debugging tasks include creating debugging tasks, modifying debugging tasks, deploying debugging tasks, and deleting debugging tasks.
[0085] Specifically, create a debugging task to create a task for testing and debugging on the interface or through file configuration. Create a publishing task to create a task for actual publishing and service provision on the interface or through file configuration. Modify a debugging task to modify and adjust an existing debugging task. Modify a publishing task to modify and adjust an existing publishing task. Deploy a debugging task to deploy a debugging task to a test environment for execution. Deploy a publishing task to deploy a publishing task to a production environment for execution. Delete a debugging task to delete a debugging task that is no longer needed. Delete a publishing task to delete a publishing task that is no longer needed. You can choose to configure Create a Publishing Task and Create a Debug Task on the interface or in a file.
[0086] It should be noted that the model generation system in the embodiment of the present application can also be connected to the feature platform to facilitate continuous optimization of the target model through new data sets continuously generated by the feature platform.
[0087] By obtaining the model training data set, determining the standardized computing environment of the target model to be generated, debugging the target model to be generated based on the standardized computing environment and target training data, the target model is automatically generated based on the standardized computing environment, unifying the algorithm and software version, so that the generated target model has a standard computing environment, facilitating the maintenance of the target model and improving the maintainability of the model. In addition, by automatically generating the target model, manual intervention is reduced, and the generation efficiency and reliability of the target model are improved.
[0088] In an optional embodiment, the above method further includes:
[0089] The training process of the target model to be generated is monitored in real time so that the model trainer can view the running data of the task in real time and visualize the training process.
[0090] Specifically, the model training module in the model generation system also includes visualization tools such as tensorflow to display the training process of the target model to be generated in real time, making it easier for model trainers to view the training results.
[0091] It should be noted that during the debugging of the target model to be generated, the model trainer can also deploy task logs, debug task logs, and view task logs.
[0092] In an optional embodiment, the above method further includes:
[0093] A software library is provided through a distributed technology framework module; the software library includes a standardized computing environment for the target model to be generated.
[0094] Specifically, Figure 6 The distributed technology framework module shown is used to provide the compilation environment and model library required for online development, and to provide a data support framework for model training, such as HiveSQL, Spark, etc. The data required for model training can be exported through frameworks such as HiveSQL and Spark. The model library includes software libraries such as TensorFlow and JupyterLab that facilitate developers to develop and call model training online, providing software resources for model training.
[0095] In an optional embodiment, the above method further includes:
[0096] The cloud resource coordination module provides a container for debugging the target model to be generated.
[0097] Specifically, the cloud resource coordination module is used to provide the required data and computing resources for the data management module, online development module and model training module, and to manage the storage resources, computing resources and image resources required in the training process and provide image services.
[0098] Furthermore, the cloud resource coordination module can also provide distributed storage for the data management module to support the import, storage, export, and analysis of training data.
[0099] like Figure 7 As shown in the figure, the container orchestration platform K8s (kubernetes) provides a standardized development environment and container resources required for model training to meet the needs of resource isolation. By deploying on the host machine, the docker container on the host machine is managed, and the image's memory, CPU, GPU and other computing resources can be dynamically configured. The container can be created, analyzed, queried, exported, stopped, restarted, and other operations can be realized. By setting parameters to generate a Dockerfile, the container can be customized to provide support for model deployment and task execution.
[0100] For example, users can create and manage servers, store and manage server host information such as IP, CPU, GPU, memory, and hard disk. Different tasks will run on different servers, and each task will run in a container, so as to achieve environmental isolation and better manage container clusters. By viewing the running status of the container in real time, corresponding actions can be taken in time when a failure occurs. Containers can be created through yaml files or from a form creation page, specifying container-related resource parameters, environment variables, and related parameters for providing services to the outside world. At the same time, it is equipped with an alarm function to detect whether the container service is normal and can promptly warn container users.
[0101] like Figure 8 As shown, the embodiment of the present application provides a model generation system, the system comprising:
[0102] Data management module 801, used to obtain model training data set;
[0103] An online development module 802 is used to determine a standardized computing environment for a target model to be generated;
[0104] The model training module 803 is used to debug the target model to be generated based on the computing environment and the target training data to generate the target model; the target training data is the data in the model training data set.
[0105] In an optional embodiment, the data management module 801 is specifically used for:
[0106] Create a data table; the data table includes the data to be imported;
[0107] Import data tables into pre-built datasets;
[0108] Clean the data in the dataset to obtain the model training dataset.
[0109] In an optional embodiment, the data management module 801 is specifically used for:
[0110] Verify the data in the data set based on the custom rules to obtain a verification data set;
[0111] The data in the verification data set whose data similarity is higher than the similarity threshold are merged to obtain the model training data set.
[0112] In an optional embodiment, the model training module 803 is further used to:
[0113] Get the development image, working directory, and mount directory of the model to be imported;
[0114] The model to be imported is imported based on the development image, working directory and mounting directory of the model to be imported; the model to be imported includes the target model to be generated.
[0115] In an optional embodiment, the online development module 802 is specifically used for:
[0116] Obtaining a custom computing environment selection result for a target model to be generated;
[0117] Based on the custom computing environment selection results, determine the standardized development environment and standardized compilation environment.
[0118] In an optional embodiment, the model training module 803 is specifically used for:
[0119] Select the target model to be generated from the entered models;
[0120] In a standardized development environment and a standardized compilation environment, the target model to be generated is trained based on the target training data.
[0121] By obtaining the model training data set, determining the standardized computing environment of the target model to be generated, debugging the target model to be generated based on the standardized computing environment and target training data, the target model is automatically generated based on the standardized computing environment, unifying the algorithm and software version, so that the generated target model has a standard computing environment, facilitating the maintenance of the target model and improving the maintainability of the model. In addition, by automatically generating the target model, manual intervention is reduced, and the generation efficiency and reliability of the target model are improved.
[0122] An embodiment of the present application provides a computer storage medium for storing a computer program. When the computer program is executed, it is used to implement any one of the above-mentioned model generation methods.
[0123] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0124] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0125] The technical content provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for general technical personnel in the field, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A model generation method, characterized in that: The method comprises: Get the model training dataset; Determine a standardized computing environment for the target model to be generated; Based on the standardized computing environment and target training data, the target model to be generated is debugged to generate a target model; the target training data is the data in the model training data set.
2. The method according to claim 1, characterized in that The obtaining of the model training data set comprises: Creating a data table; the data table includes data to be imported; Importing the data table into a pre-established data set; The data in the data set is cleaned to obtain the model training data set.
3. The method according to claim 2, characterized in that The step of cleaning the data in the data set to obtain the model training data set includes: Verify the data in the data set based on the custom rules to obtain a verification data set; The data in the verification data set whose data similarity is higher than the similarity threshold are merged to obtain the model training data set.
4. The method according to claim 1, characterized in that: The method further comprises: Obtain the development image, working directory and mounting directory of the model to be entered; Based on the development image, working directory and mounting directory of the model to be imported, the model to be imported is imported; the model to be imported includes the target model to be generated.
5. The method according to claim 1, characterized in that The step of determining a standardized computing environment for a target model to be generated includes: Obtaining a custom computing environment selection result for the target model to be generated; Based on the customized computing environment selection result, a standardized development environment and a standardized compilation environment are determined.
6. The method according to claim 5, characterized in that The debugging of the target model to be generated based on the computing environment and the target training data includes: Selecting the target model to be generated from the entered models; In the standardized development environment and the standardized compilation environment, the target model to be generated is trained based on the target training data.
7. A model generation system, characterized in that: The system comprises: Data management module, used to obtain model training data sets; An online development module for determining a standardized computing environment for the target model to be generated; The model training module is used to debug the target model to be generated based on the computing environment and target training data to generate the target model; the target training data is the data in the model training data set.
8. The system according to claim 7, characterized in that The online development module is specifically used for: Obtaining a custom computing environment selection result for the target model to be generated; Based on the customized computing environment selection result, a standardized development environment and a standardized compilation environment are determined.
9. The system according to claim 8, characterized in that The model training module is specifically used for: Selecting the target model to be generated from the entered models; In the standardized development environment and the standardized compilation environment, the target model to be generated is trained based on the target training data.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the model generation method according to any one of claims 1 to 6 is implemented.