Model-based data writing method and device, computer equipment and storage medium

Abstract

The invention discloses a model-based data writing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a plurality of preposed data tables through a first preset script program; after a preset prediction model is loaded through a preset target loading program, predicting a plurality of rear data tables from the plurality of front data tables; acquiring a plurality of pieces of post-data from the post-data table through a second preset script program; and writing the plurality of pieces of post data into a preset data table according to a preset writing rule. According to the scheme, data processing and model decoupling are performed by using the preset script program and the preset prediction model respectively, so that the business logic coupling degree is reduced; when modification and replacement of the model and updating of the configuration information occur, the process is achieved after loading through a preset target loading program, and the iteration period is prolonged; by writing the obtained post data into the preset data table, all final data can be summarized into the same data table, the standard is unified, and the accuracy and efficiency of data analysis are improved.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent decision-making and financial technology, and in particular to a model-based data writing method, device, computer equipment and storage medium. Background Art

[0002] In the existing architecture, data processing and model prediction functions are tightly coupled to form an intricate whole. This architecture is applied in a variety of scenarios. For example, in the actual financial business scenario, this architecture will formulate two or more business plans based on the same optimized business goal and analyze which business plan meets expectations. This design approach brings multiple challenges: 1. Since business requirements are deeply bound to the codes of data processing and model prediction, any minor adjustments in the business may require developers to find and modify specific parts in the huge code base, which not only increases the complexity of modification, but may also lead to new errors due to unfamiliarity with code details; 2. Under the existing architecture, when a model needs to be updated or the configuration needs to be adjusted, it is usually necessary to stop the currently running task and complete the system version release process, which not only prolongs the time to respond to needs and reduces system availability, but also increases the potential risks caused by version updates. In addition, since each update may involve a large number of code changes, it is difficult to quickly and flexibly adapt to changing business needs; 3. The existing architecture is often not designed with full consideration of future expansion needs, resulting in the need to develop new code branches for each expansion. This not only increases the complexity of the system, but also makes subsequent maintenance and data analysis particularly difficult. In addition, due to the lack of unified standards and specifications for data tables and fields, data analysts need to spend a lot of time learning the structure of different tables and distinguishing the meaning of fields, which reduces the efficiency and accuracy of data analysis.

[0003] Therefore, those skilled in the art are in urgent need of finding a new technical solution to solve the above-mentioned technical problems. Summary of the invention

[0004] Based on this, a model-based data writing method, device, computer equipment and storage medium are provided to solve the technical problems in the prior art that the architecture has serious coupling between data processing and model prediction, long iteration cycle and is not conducive to data analysis.

[0005] A data writing method based on a model, comprising: Acquire multiple front-end data tables through a first preset script program; After loading the preset prediction model through the preset target loading program, predicting multiple post-data tables from the multiple pre-data tables; Acquire a plurality of post-data from the post-data table through a second preset script program; The plurality of post-data are written into a preset data table according to a preset writing rule.

[0006] A data writing device based on a model, comprising: A first acquisition module, used for acquiring a plurality of pre-data tables through a first preset script program; A prediction module, used to predict a plurality of post-data tables from a plurality of the pre-data tables after loading a preset prediction model through a preset target loading program; A second acquisition module, used for acquiring a plurality of post-data from the post-data table through a second preset script program; The writing module is used to write the plurality of post-data into a preset data table according to a preset writing rule.

[0007] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned model-based data writing method when executing the computer program.

[0008] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the model-based data writing method is implemented.

[0009] The above-mentioned model-based data writing method, device, computer equipment and storage medium obtain multiple front data tables through a first preset script program; after loading a preset prediction model through a preset target loading program, multiple rear data tables are predicted from the multiple front data tables; and multiple rear data are obtained from the rear data tables through a second preset script program; The plurality of post-data are written to the preset data table according to the preset writing rules. This solution decouples data processing and model by using preset script programs and preset prediction models respectively, reduces the coupling degree of business logic, and avoids the introduction of new errors; when the model is modified and replaced and the configuration information is updated, the modification, replacement and update are realized after loading by the preset target loader, which improves the iteration cycle and reduces the potential risks brought by the update, and can be applied to the ever-changing business needs; the post-data is obtained from the post-data table by the second preset script program and then written to the preset data table, so that all the final data can be aggregated into the same data table, unifying the standards and specifications in the data table, and at the same time, for the later data analysis, it reduces the difficulty of data normalization and improves the accuracy and efficiency of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 is a schematic diagram of an application environment of a model-based data writing method in an embodiment of the present invention; Figure 2 is a process diagram of a model-based data writing method in one embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a model-based data writing device in one embodiment of the present invention; Figure 4 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The model-based data writing method provided by the present invention can be applied in Figure 1 In an application environment, a client communicates with a server through a network. The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented as an independent server or a server cluster consisting of multiple servers, such as the present invention may be applied to the server corresponding to the data middle station.

[0014] In one embodiment, if Figure 2 As shown, a model-based data writing method is provided, which is applied in Figure 1 The server in the example is used as an example to illustrate the process, including the following steps: S10, obtaining a plurality of pre-set data tables through a first preset script program.

[0015] It is understandable that the first preset script program can be a pre-written script program, and the written script program can connect to the corresponding database or data table to directly obtain the corresponding front-end data table. The connection process requires the database connection library to be installed in advance. For example, for Python, the database connection library needs to be installed including pymysql (for MySQL) or psycopg2 (for PostgreSQL). You can use pip to install the database connection library, such as pip install pymysql.

[0016] S20, after loading the preset prediction model through the preset target loading program, predicting multiple post-data tables from the multiple pre-data tables.

[0017] It can be understood that the preset target loader can be a Spark program, which can load the preset prediction model from the hdfs file system and load the configuration information from the mysql database. After obtaining the above-mentioned preset prediction model, multiple post-data tables predicted by the model can be directly obtained in the configuration information, wherein the predicted post-data tables already have different scenario attributes, such as the post-data tables corresponding to the experimental group and the test group.

[0018] S30, obtaining a plurality of post-data from the post-data table through a second preset script program.

[0019] It can be understood that the second preset script program can be a pre-written script program, and the written script program can also directly obtain multiple post-data from the connected post-data table (for subsequent writing to the data table). The obtained post-data can be obtained according to the acquisition rules corresponding to the script program. In this way, the obtained post-data can be numerical, character or other types of data. The extracted post-data will be further processed and verified to ensure its accuracy and completeness. The means of processing and verification may include data cleaning, format conversion or comparison with other data sources and other operations.

[0020] S40, writing the plurality of post-data into a preset data table according to a preset writing rule.

[0021] Understandably, the preset writing rule may be a preset writing field type, such as writing according to the field types of general field 1, general field 2... non-general field (json), business name, tag, model name, and the format of the preset data table formed is also the same, wherein general field 1, general field 2, ... fields may be used to store general information required across multiple businesses or data models, such as timestamp, user ID, status code, etc., non-general field (json) may be used to store unstructured information specific to a certain business or data model, various types of data may be flexibly stored using the JSON format, such as nested objects, arrays, etc., and when processing fields in the JSON format, it is necessary to parse the JSON string to extract the required information, the business name field may be used to identify the business or module to which the data belongs, and is helpful in distinguishing data of different businesses during data processing and storage, the tag field may be used to classify or mark the data, specifically for screening, filtering or aggregating data, the model name field may be used to identify the machine learning model used to process the data, and when loading and applying the model, it is necessary to know the name of the model in order to load and use it correctly, and it may also be associated with the model configuration or model metadata.

[0022] It should be noted that the embodiment of steps S10 to S40 can be applied to financial business scenarios, that is, by formulating two or more business plans for the same optimization goal, dividing different users into different groups, and using statistical methods to collect user feedback data to determine which business plan is better.

[0023] In the embodiment of step S10 to step S40, by using a preset script program and a preset prediction model respectively, data processing and model are decoupled, the business logic coupling is reduced, and the introduction of new errors is avoided; when the model is modified and replaced and the configuration information is updated, the modification, replacement and update are realized after loading by the preset target loader, thereby improving the iteration cycle, reducing the potential risks brought by the update, and being applicable to changing business needs; by obtaining the post-data from the post-data table through the second preset script program and writing it to the preset data table, all the final data can be aggregated into the same data table, unifying the standards and specifications in the data table, and at the same time, for the later data analysis, the difficulty of data normalization is reduced, and the accuracy and efficiency of data analysis are improved.

[0024] Furthermore, the obtaining of a plurality of pre-set data tables through a first preset script program includes: Obtaining data table connection parameters corresponding to the first preset script program; After the first preset script program is connected to the preset data table based on the data table connection parameters, a plurality of pre-set data tables corresponding to the data tags are acquired through the preset query statements in the first preset script program.

[0025] Understandably, the data table connection parameters may include a host name, a port, a user name, a password, and a database name, etc.; the process of the first preset script program connecting to the preset data table in the database is to automatically access the database based on the data table connection parameters through the language in the script, query through the preset query statement in the script, and return the queried data in the format of a preset front-end data table, wherein the preset query statement may be used, for example, query= f"SELECT * FROM {table} in a Python script, where table is a preset data table in the data table; This embodiment connects to the preset data table in the database through the first preset script program, which can automatically obtain the corresponding front data table and use the front data table for pre-processing before model prediction. At the same time, in the acquisition process, the security of data acquisition can be improved by connecting and then acquiring.

[0026] Furthermore, a plurality of pre-placed data tables corresponding to the data tags are obtained through the preset query statements in the first preset script program, including: Obtain a pre-data table differentiated by data usage scenario attributes; one usage scenario attribute corresponds to at least one data tag; When the usage scenario attribute matches the data tag, a plurality of the front-end data tables corresponding to the plurality of the data tags are acquired through a preset query statement in the first preset script program.

[0027] Understandably, the scenario attribute may refer to the scenario in which the data needs to be used, such as a scenario for an experiment, a scenario for a control, and a scenario for a test; Specifically, the corresponding data label exists in the queried front data table, that is, in the process of obtaining multiple front data tables, the front data in each front data table obtained by the script program will be marked with data labels in the corresponding matching scene attributes (in the query process, the scene attributes can be determined by obtaining the attributes of the data in the original preset data table), such as the front data in the same front data table has four data labels: experimental group 1, experimental group 2, control group 1, and control group 2; The pre-data table with data tags obtained in this embodiment can be directly obtained in the next step of the preset target loading program, thereby improving the processing efficiency of the preset target loading program, and at the same time realizing the decoupling of data processing and model prediction.

[0028] Furthermore, after the preset prediction model is loaded by the preset target loading program, multiple post-data tables are predicted from the multiple pre-data tables, including: Storing the configuration information into the variables related to the preset prediction model through the preset target loading program; Storing the model object into the variable related to the preset prediction model through the preset target loading program; Acquire multiple front-end data tables corresponding to the configuration information through the variables and determine the preset prediction model; After predicting, classifying and aggregating the plurality of the front-end data tables through the preset prediction model, a plurality of the rear-end data tables are obtained.

[0029] Understandably, the variable may refer to a broadcast variable or a global variable of a preset target loader, which is shared by nodes or used for subsequent data processing; Specifically, after using the preset target loader Spark's JdbcRDD or DataFrameReader to connect to the MySQL database through JDBC, obtain the required configuration information, such as database connection information, table name, field mapping, etc., and store the configuration information in the preset target loader's broadcast variables so that it can be shared on each node in the cluster; use the file system API corresponding to the preset target loader (such as the FileSystem class) or the spark.read.textFile method of the preset target loader to read the model file from HDFS. If the preset prediction model is a pre-trained machine learning model (such as using Spark When a model trained by MLlib is used, it is loaded using methods such as PipelineModel.load, and the model object is stored in the broadcast variable or global variable of the preset target loader for use in subsequent data processing; the configuration information loaded from the MySQL database is determined by the variables, and the corresponding database is connected to the DataFrameReader of the preset target loader through JDBC, and the front data table is loaded as a DataFrame (if the front data table is distributed in multiple databases, it is necessary to connect and load the front data table in each database separately); the preset prediction model corresponding to the model file on HDFS is determined by the variables; the loaded model is applied to the filtered front data table for prediction, classification, clustering and other processing to obtain multiple post-data tables, wherein, during the prediction process, the front data in the front data table can be predicted; This embodiment uses the parallel processing capabilities and optimization strategies (such as partitioning, caching, etc.) of the preset target loader to improve performance when processing large data sets, and improves data security through the connected system or database when loading configuration information, predictive prediction models, and front-end data tables.

[0030] Furthermore, before the obtaining of a plurality of pre-set data corresponding to the configuration information through the variables and determining the preset prediction model, the method further includes: Extract and transform the preset training data to obtain target training data; The target training data is trained based on a preset clustering algorithm, and when the clustering effect after training meets the expected clustering effect, the preset prediction model is obtained.

[0031] Understandably, the process of using the preset prediction model after training can be divided into the following steps for processing, including: 1. Data preparation; 2. Model selection; 3. Model training; 4. Model evaluation; 5. Model saving and loading; 6. Model deployment and application; Specifically, preset training data is collected from various data sources (such as databases, files, networks, etc.), and data cleaning, data conversion and feature engineering are performed on the preset training data to obtain target training data. Among them, data cleaning handles missing values, outliers, duplicate data, etc. to ensure data quality. Data conversion converts data into a format suitable for model training, which may include feature selection, feature scaling, encoding, etc. Feature engineering builds and selects features according to business needs and data characteristics to improve model performance; determine the appropriate machine learning algorithm according to the problem type (such as classification, regression, clustering, etc.), including linear regression, logistic regression, decision tree, random forest, support vector machine, clustering algorithm (such as K-means), etc.: use the provided API to create an instance of the selected algorithm, based on Set the hyperparameters of the preset prediction model according to data characteristics and business needs, such as the number of iterations, learning rate, regularization parameters, etc., and use the target training data to fit the model, that is, execute the algorithm optimization process to find the best parameter combination; select appropriate evaluation indicators according to the problem type, such as accuracy, recall rate, F1 score, mean square error, etc., and use cross-validation techniques (such as K-fold cross-validation) to evaluate the generalization ability of the model; adjust model parameters or feature engineering according to the evaluation results (the clustering effect after training meets the expected clustering effect) to improve model performance; save the trained model to the file system or distributed storage system for subsequent use; when using it, use the API provided by the preset target loader to load the saved model; deploy the model to the production environment, such as Spark Streaming application, Web service, database, etc.; The preset prediction model trained in this embodiment is used in the process of data prediction. When combined with data processing, multiple front data tables can be split to obtain multiple rear data tables, and then the front data in the front data table can be further processed to obtain a rear data table with higher division accuracy.

[0032] Furthermore, the step of writing the plurality of post-data into a preset data table according to a preset writing rule includes: Obtaining a field type corresponding to the preset writing rule; The field attributes of the plurality of post-data are split according to the field type and then written into the preset data table.

[0033] Understandably, the field type can be the above-mentioned general field 1, general field 2... non-general field (json), business name, tag, model name; each field type is written into the corresponding format in the preset data table according to the specified field attributes matching the field type to form the final preset data table, in which there is a mapping matching relationship between the field attributes and the field type; This embodiment can ensure the accuracy and consistency of the data by splitting and writing the post-data strictly according to the preset field types, which reduces errors and exceptions caused by data format errors or type mismatches; by writing to the preset data table in the specified format, data analysts have clear standards and specifications, and do not need to spend a lot of time learning the structure of different data tables, thereby improving the efficiency and accuracy of subsequent data analysis by data personnel.

[0034] Furthermore, after writing the plurality of post-data into the preset data table according to the preset writing rule, the method further includes: Get the filtered field type; The preset data table is screened according to the field type and the post-data of the screened field type is subjected to data normalization processing to obtain a data normalization result.

[0035] Understandably, each field type in the preset data table can be used as the basis for data screening, and data screening can also be performed through multiple field types; data normalization refers to a processing method that scales data to a specific range (usually [0,1] or [-1,1]), and its main purpose is to eliminate the dimensional differences between features. The available methods include minimum-maximum normalization, Z-score standardization, decimal normalization, data type normalization, format normalization, clipping normalization, and standard deviation normalization. The method can be a business-related analysis method. The analysis of the data normalization results can analyze the uniformity of the data to avoid extreme values ​​or outliers, and can analyze data with obvious outliers; in addition, the data normalization results after data analysis can be presented in a visual and obvious form; This embodiment can perform data aggregation and analysis of different businesses in the same preset data table, greatly improving the accuracy and normalization of the analysis.

[0036] In summary, the above provides a model-based data writing method, which obtains multiple front data tables through a first preset script program; after loading the preset prediction model through a preset target loader, predicts multiple post data tables from the multiple front data tables; obtains multiple post data from the post data table through a second preset script program; and writes the multiple post data to the preset data table according to the preset writing rules. In this way, the scheme decouples data processing and model by using preset script programs and preset prediction models respectively, reduces the coupling degree of business logic, and avoids introducing new errors; when the model is modified and replaced and the configuration information is updated, the modification, replacement and update are realized after loading through the preset target loader (equivalent to scalability, specifically, if a model needs to be offline, it is only necessary to stop the task, and there is no need to modify the code and other work that requires versioning. If a new model needs to be launched, it is only necessary to launch a new task and configure the information, without modifying the code. In the whole process, the processes of each business are isolated, and the online and offline operations can be independent, and even multiple people can do it at the same time). No need to release a version, which improves the iteration cycle, reduces the potential risks brought by updates, and can be adapted to ever-changing business needs. By obtaining the post-data from the post-data table through the second preset script program and writing it to the preset data table, all the final data can be summarized in the same data table, unifying the standards and specifications in the data table. At the same time, for later data analysis, it reduces the difficulty of data normalization and improves the accuracy and efficiency of data analysis. In addition, when model data needs to be merged or split, it is only necessary to modify the first preset script program for front-end data processing to merge data labels or split data labels. The process of replacing the script does not require release, which can also improve the iteration cycle of business needs.

[0037] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0038] In one embodiment, a model-based data writing device is provided, and the model-based data writing device corresponds one-to-one to the model-based data writing method in the above embodiment. Figure 3 As shown, the model-based data writing device includes a first acquisition module 11, a prediction module 12, a second acquisition module 13 and a writing module 14. The functional modules are described in detail as follows: A first acquisition module 11, used to acquire a plurality of pre-data tables through a first preset script program; A prediction module 12, configured to predict a plurality of post-data tables from a plurality of the pre-data tables after loading a preset prediction model through a preset target loading program; A second acquisition module 13, used for acquiring a plurality of post-data from the post-data table through a second preset script program; The writing module 14 is used to write the plurality of post-data into a preset data table according to a preset writing rule.

[0039] Furthermore, the first acquisition module includes: A first acquisition unit, used to acquire data table connection parameters corresponding to the first preset script program; A second acquisition unit is configured to acquire a plurality of pre-set data tables corresponding to data tags through a preset query statement in the first preset script program after the first preset script program is connected to the preset data table based on the data table connection parameters.

[0040] Furthermore, the second acquiring unit includes: A first acquisition subunit is used to acquire a pre-data table differentiated according to data usage scenario attributes; one usage scenario attribute corresponds to at least one data tag; The second acquisition subunit is used to acquire a plurality of the front data tables corresponding to the plurality of the data tags through a preset query statement in the first preset script program when the usage scenario attribute matches the data tag.

[0041] Furthermore, the prediction module includes: A first storage unit, used to store configuration information into variables related to the preset prediction model through the preset target loading program; A second storage unit, used to store the model object into the variable related to the preset prediction model through the preset target loading program; A determination unit, configured to obtain a plurality of pre-data tables corresponding to the configuration information through the variables and determine the preset prediction model; A processing unit is used to obtain a plurality of post-data tables after predicting, classifying and aggregating the plurality of pre-data tables through the preset prediction model.

[0042] Furthermore, the model-based data writing device further includes: An extraction and conversion module is used to extract and convert features of preset training data to obtain target training data; The training module is used to train the target training data based on a preset clustering algorithm, and obtain the preset prediction model when the clustering effect after training meets the expected clustering effect.

[0043] Furthermore, the writing module includes: A third acquisition unit, used to acquire a field type corresponding to the preset writing rule; A writing unit is used to split the field attributes of the plurality of post-data according to the field type and write the split attributes into the preset data table.

[0044] Furthermore, the model-based data writing device further includes: The third acquisition module is used to obtain the filtered field type; The data normalization processing module is used to filter the preset data table according to the field type and perform data normalization processing on the post-data of the filtered field type to obtain a data normalization result.

[0045] For the specific definition of the model-based data writing device, please refer to the definition of the model-based data writing method above, which will not be repeated here. Each module in the above-mentioned model-based data writing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0046] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the model-based data writing method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model-based data writing method is implemented.

[0047] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the model-based data writing method in the above embodiment are implemented, such as Figure 2 Steps S10 to S40 shown include the following steps: Acquire multiple front-end data tables through a first preset script program; After loading the preset prediction model through the preset target loading program, predicting multiple post-data tables from the multiple pre-data tables; Acquire a plurality of post-data from the post-data table through a second preset script program; The plurality of post-data are written into a preset data table according to a preset writing rule.

[0048] Alternatively, when the processor executes the computer program, the functions of each module / unit of the model-based data writing device in the above embodiment are realized, for example Figure 3 The functions of the modules 14 to 13 are shown in Figure 1. To avoid repetition, they will not be described here.

[0049] In one embodiment, a computer-readable storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, the steps of the model-based data writing method in the above embodiment are implemented, such as Figure 2 Steps S10 to S40 shown include the following steps: Acquire multiple front-end data tables through a first preset script program; After loading the preset prediction model through the preset target loading program, predicting multiple post-data tables from the multiple pre-data tables; Acquire a plurality of post-data from the post-data table through a second preset script program; The plurality of post-data are written into a preset data table according to a preset writing rule.

[0050] Alternatively, when the computer program is executed by the processor, the functions of each module / unit of the model-based data writing device in the above-mentioned embodiment are realized, for example Figure 3 The functions of modules 11 to 14 are shown in Figure 1. To avoid repetition, they will not be described here.

[0051] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0052] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0053] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data writing method based on a model, characterized in that: include: Acquire multiple front-end data tables through a first preset script program; After loading the preset prediction model through the preset target loading program, predicting multiple post-data tables from the multiple pre-data tables; Acquire a plurality of post-data from the post-data table through a second preset script program; The plurality of post-data are written into a preset data table according to a preset writing rule.

2. The model-based data writing method according to claim 1, characterized in that: The method of obtaining a plurality of pre-set data tables through a first preset script program includes: Obtaining data table connection parameters corresponding to the first preset script program; After the first preset script program is connected to the preset data table based on the data table connection parameters, a plurality of pre-set data tables corresponding to the data tags are acquired through the preset query statements in the first preset script program.

3. The model-based data writing method according to claim 2, characterized in that: The method of obtaining a plurality of pre-set data tables corresponding to the data tags through the preset query statement in the first preset script program includes: Obtain a pre-data table differentiated by data usage scenario attributes; one usage scenario attribute corresponds to at least one data tag; When the usage scenario attribute matches the data tag, a plurality of the front-end data tables corresponding to the plurality of the data tags are acquired through a preset query statement in the first preset script program.

4. The model-based data writing method according to claim 1, characterized in that: After the preset prediction model is loaded by the preset target loading program, a plurality of post-data tables are predicted from the plurality of pre-data tables, including: Storing the configuration information into the variables related to the preset prediction model through the preset target loading program; The model object is stored in the variable related to the preset prediction model by the preset target loading program; Acquire multiple front-end data tables corresponding to the configuration information through the variables and determine the preset prediction model; After the plurality of the front-end data tables are predicted, classified and aggregated by the preset prediction model, a plurality of the rear-end data tables are obtained.

5. The model-based data writing method according to claim 4, characterized in that: Before obtaining a plurality of pre-set data corresponding to the configuration information through the variables and determining the preset prediction model, the method further includes: Extract and transform the preset training data to obtain target training data; The target training data is trained based on a preset clustering algorithm, and when the clustering effect after training meets the expected clustering effect, the preset prediction model is obtained.

6. The model-based data writing method according to claim 1, characterized in that: The step of writing the plurality of post-data into a preset data table according to a preset writing rule comprises: Obtaining a field type corresponding to the preset writing rule; The field attributes of the plurality of post-data are split according to the field type and then written into the preset data table.

7. The model-based data writing method according to claim 1, characterized in that: After the plurality of post-data are written into the preset data table according to the preset writing rule, the method further includes: Get the filtered field type; The preset data table is screened according to the field type and the post-data of the screened field type is subjected to data normalization processing to obtain a data normalization result.

8. A data writing device based on a model, characterized in that: include: A first acquisition module, used for acquiring a plurality of pre-data tables through a first preset script program; A prediction module, used to predict a plurality of post-data tables from a plurality of the pre-data tables after loading a preset prediction model through a preset target loading program; A second acquisition module, used for acquiring a plurality of post-data from the post-data table through a second preset script program; The writing module is used to write the plurality of post-data into a preset data table according to a preset writing rule.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the model-based data writing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model-based data writing method according to any one of claims 1 to 7 is implemented.

Images