Network model publishing method and device, storage medium, and electronic device

Abstract

The present invention discloses a method and device for publishing a network model, a storage medium, and an electronic device, and belongs to the field of artificial intelligence. The method includes: obtaining offline sample data of the current cycle according to the file directory of the target device; generating a training sample wide table according to the output duration of the offline sample data, wherein the training sample wide table is used to store sample data according to time slices; splitting the training sample wide table into a training set and a verification set; using the training set and the verification set on the target device to iteratively update the historical network model of the previous cycle, and publishing the trained target network model. The present invention solves the technical problem of the long cycle of publishing network models in related technologies, reduces dependence on business links, and improves the flexibility of the system.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for publishing a network model, a storage medium, and an electronic device. Background Art

[0002] In related technologies, in order to achieve online prediction, it is necessary to use a large amount of sample data to train the network model in advance.

[0003] In related technologies, the automated training and deployment of models in various scenarios often use a specific scheduling system to schedule scripts such as feature engineering, sample generation, and model training. The entire process involves the conversion of multiple devices, multiple programming languages, multiple systems, and multiple data storage methods. It is cumbersome and complicated to use. In addition, data security and risk control in user information-sensitive fields such as finance are particularly strict. Data is often impossible to migrate to other devices for personalized calculations and recommendations. Therefore, the automated training and deployment of models based on native systems is particularly difficult. The producers of network models can only use data provided by third parties as samples, and then perform a series of data preprocessing and filtering, resulting in a long release cycle for network models.

[0004] With respect to the above-mentioned problems existing in the related technologies, no effective solutions have been found so far. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for publishing a network model, a storage medium, and an electronic device.

[0006] According to one aspect of an embodiment of the present application, a method for publishing a network model is provided, comprising: obtaining offline sample data of a current period according to a file directory of a target device; generating a training sample wide table according to the output duration of the offline sample data, wherein the training sample wide table is used to store sample data according to time slices; splitting the training sample wide table into a training set and a validation set; iteratively updating a historical network model of the previous period on the target device using the training set and the validation set, and publishing the trained target network model.

[0007] Furthermore, obtaining offline sample data of the current period according to the file directory of the target device includes: determining the last settlement date of the period before the current period; searching for the file directory storing the last settlement date in a preset database of the target device; and generating offline sample data of the current period according to the file directory.

[0008] Further, searching for the file directory stored on the last settlement date in the preset database of the target device includes: using a child process to call a hadoop command in the target device; executing the hadoop command, and determining whether a Hadoop distributed file system HDLS path of the last settlement date exists in the preset database; if the HDLS path of the last settlement date exists in the preset database, determining the data under the corresponding path as the file directory stored on the last settlement date.

[0009] Furthermore, generating offline sample data of the current period according to the file directory includes: extracting feature items in the file directory, wherein each feature item corresponds to a feature dimension of the target sample; associating the feature items to the feature columns of the feature table through the sample ID, and splicing to generate an offline feature wide table of the current period.

[0010] Furthermore, generating a wide table of training samples according to the production duration of the offline sample data includes: determining the end date and start date of the current cycle according to the production duration of the offline sample data; reading offline data of the day before the start date from the offline sample data, and processing and generating offline feature data; cyclically reading full data of each day from the start date to the end date from the offline sample data, and processing and generating sample feature data; cyclically reading online data of each day from the start date to the end date from the offline sample data, and processing and generating online feature data; associating the offline feature data, the sample feature data, and the online feature data according to date and sample ID to generate the wide table of training samples.

[0011] Furthermore, publishing the trained target network model includes: converting the model file of the trained target network model into an ONNX model using the open neural network exchange ONNX format; and publishing the ONNX model.

[0012] Furthermore, after publishing the trained target network model, the method also includes: responding to the prediction request of the target network model online, obtaining the online feature data carried in the prediction request; using the online feature data to generate input feature data based on the offline sample data; inputting the input feature data into the target network model, and outputting the prediction result online.

[0013] According to another aspect of an embodiment of the present application, a network model publishing device is also provided, including: an acquisition module, used to obtain offline sample data of the current period according to the file directory of the target device; a first generation module, used to generate a training sample wide table according to the output duration of the offline sample data, wherein the training sample wide table is used to store sample data according to time slices; a splitting module, used to split the training sample wide table into a training set and a verification set; a publishing module, used to iteratively update the historical network model of the previous period on the target device using the training set and the verification set, and publish the trained target network model.

[0014] Furthermore, the acquisition module includes: a determination unit, used to determine the last settlement date of the previous cycle of the current cycle; a search unit, used to search for the file directory storing the last settlement date in the preset database of the target device; and a generation unit, used to generate offline sample data of the current cycle based on the file directory.

[0015] Further, the search unit includes: a calling subunit, used to use a subprocess to call a hadoop command in the target device; a judging subunit, used to execute the hadoop command and judge whether there is a Hadoop distributed file system HDLS path of the last settlement date in the preset database; and a determining subunit, used to determine the data under the corresponding path as the file directory stored on the last settlement date if the HDLS path of the last settlement date exists in the preset database.

[0016] Furthermore, the generation unit includes: an extraction subunit, used to extract feature items in the file directory, wherein each feature item corresponds to a feature dimension of the target sample; a splicing subunit, used to associate the feature items with the feature columns of the feature table through the sample ID, and splice to generate an offline feature wide table of the current cycle.

[0017] Furthermore, the first generation module includes: a determination unit, which is used to determine the end date and start date of the current cycle according to the output duration of the offline sample data; a processing unit, which is used to read the offline data of the day before the start date from the offline sample data, and process and generate offline feature data; cyclically read the full data of each day from the start date to the end date from the offline sample data, and process and generate sample feature data; cyclically read the online data of each day from the start date to the end date from the offline sample data, and process and generate online feature data; a generation unit, which is used to associate the offline feature data, the sample feature data, and the online feature data according to the date and sample ID to generate the training sample wide table.

[0018] Furthermore, the publishing module includes: a conversion unit, which is used to convert the model file of the trained target network model into an ONNX model using the open neural network exchange ONNX format; and a publishing unit, which is used to publish the ONNX model.

[0019] Furthermore, the device also includes: a response module, which is used to respond to the prediction request of the target network model online after the publishing module publishes the trained target network model, and obtain the online feature data carried in the prediction request; a second generation module, which is used to use the online feature data to generate input feature data based on the offline sample data; and a prediction module, which is used to input the input feature data into the target network model and output the prediction result online.

[0020] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.

[0021] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.

[0022] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.

[0023] Through the present invention, offline sample data of the current period is obtained according to the file directory of the target device, and a training sample wide table is generated according to the output duration of the offline sample data. The training sample wide table is used to store sample data according to time slices, and the training sample wide table is split into a training set and a verification set. The training set and the verification set are used to iteratively update the historical network model of the previous period on the target device, and the trained target network model is published. By using the file directory of the target device to obtain and generate the training sample wide table, and then training and publishing the updated network model in the target device, a solution for automatic training and deployment of network models for devices in a business native system is implemented, which avoids the migration and conversion of business data between multiple devices and systems, reduces the security risks of sensitive data, solves the technical problem of long cycle of publishing network models in related technologies, reduces dependence on business links, and improves the flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 is a hardware structure block diagram of a server according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of a method for publishing a network model according to an embodiment of the present invention;

[0027] Figure 3 is a flow chart of automatic feature processing in an embodiment of the present invention;

[0028] Figure 4 is a flow chart of generating a wide table of training samples according to an embodiment of the present invention;

[0029] Figure 5 is a flow chart of an automatic training model according to an embodiment of the present invention;

[0030] Figure 6 is a flow chart of an automated publishing model according to an embodiment of the present invention;

[0031] Figure 7 is a flow chart of offline training and online publishing of a model according to an embodiment of the present invention;

[0032] Figure 8 is a structural block diagram of a publishing device for a network model according to an embodiment of the present invention;

[0033] Fig. 9 It is a structural block diagram of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] Example 1

[0037] The method embodiment provided in the first embodiment of the present application can be executed in a server, a computer, a mobile phone, or a similar computing device. Taking running on a server as an example, Figure 1 FIG. 1 is a hardware structure diagram of a server according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0038] The memory 104 can be used to store server programs, for example, software programs and modules of application software, such as a server program corresponding to a method for publishing a network model in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] In this embodiment, a method for publishing a network model is provided. Figure 2 is a flow chart of a method for publishing a network model according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0041] Step S202, obtaining offline sample data of the current period according to the file directory of the target device;

[0042] In this embodiment, the target device is used to load the network model, provide the front-end with API calls and display pages, and store the data generated by the network model. Therefore, sample data processing and iterative model training can be performed directly in the target device. The network model can be applied in various scenarios, such as online material recommendations, stock predictions, etc.

[0043] Step S204, generating a training sample wide table according to the output duration of the offline sample data, wherein the training sample wide table is used to store the sample data according to time slices;

[0044] In this embodiment, the output time of offline sample data corresponds to the verification time of the prediction results output by the network model in the previous cycle. The wide table associates the indicators, dimensions, and attributes related to the business subject in a database table, and stores different contents in the same table. Each table item in the wide table corresponds to a feature item of the sample and is stored by time segmentation.

[0045] Step S206, splitting the training sample wide table into a training set and a validation set;

[0046] Step S208, iteratively update the historical network model of the previous cycle using the training set and the validation set on the target device, and publish the trained target network model.

[0047] Optionally, after the target network model is released, the target device can continue to call the updated target network model for prediction and generate sample data for the next cycle.

[0048] Through the above steps, the offline sample data of the current period is obtained according to the file directory of the target device, and a training sample wide table is generated according to the output duration of the offline sample data. The training sample wide table is used to store sample data according to time slices, and the training sample wide table is split into a training set and a validation set. The training set and the validation set are used to iteratively update the historical network model of the previous period on the target device, and the trained target network model is published. The training sample wide table is obtained and generated by using the file directory of the target device, and then the updated network model is trained and published in the target device. A solution for automatic training and deployment of network models for devices in business native systems is implemented, which avoids the migration and conversion of business data between multiple devices and systems, reduces the security risks of sensitive data, solves the technical problem of long cycle of network model publishing in related technologies, reduces dependence on business links, and improves the flexibility of the system.

[0049] In one implementation of this embodiment, obtaining offline sample data of the current period according to the file directory of the target device includes:

[0050] S11, determining the last settlement date of the previous cycle of the current cycle;

[0051] S12, searching the preset database of the target device for the file directory stored with the last settlement date;

[0052] In one implementation of the present embodiment, searching for a file directory storing the last settlement date in a preset database of a target device includes: using a child process to call a hadoop command in the target device; executing the hadoop command, and determining whether a Hadoop Distributed File System (HDLS) path of the last settlement date exists in the preset database; if the HDLS path of the last settlement date exists in the preset database, determining the data under the corresponding path as the file directory storing the last settlement date.

[0053] S13, generating offline sample data of the current cycle according to the file directory.

[0054] In one implementation of this embodiment, generating offline sample data for the current period based on the file directory includes: extracting feature items in the file directory, where each feature item corresponds to a feature dimension of the target sample; associating the feature items to the feature columns of the feature table through the sample ID, and splicing to generate an offline feature wide table for the current period.

[0055] Figure 3It is a flow chart of feature automatic processing in an embodiment of the present invention, and the meanings of relevant variables are, todayDate: program running date, lstDate (lastDate): the date of the day before the program running date, lstDate1stMonth (last Date last Month): the last day of the month before the program running date. The process includes two branches, which correspond to monthly feature automatic processing and daily feature automatic processing respectively. Taking the settlement cycle as daily settlement as an example, it includes: starting the scheduled task; obtaining the date of the day; obtaining the date of the previous day; judging whether a directory named lastDate exists; if it exists, performing daily feature processing, and appending the feature wide table to the corresponding directory (ID + feature column) with the directory lastDate, otherwise, waiting for the next cycle to execute the scheduled task again.

[0056] In certain fields such as finance, data about users and cards is often stored in month-end slices. Therefore, monthly features can extract useful features from these monthly slices and store them in the same month-end slices.

[0057] In one implementation of the present embodiment, generating a wide table of training samples according to the production duration of offline sample data includes: determining the end date and start date of the current cycle according to the production duration of the offline sample data; reading offline data of the day before the start date from the offline sample data, and processing and generating offline feature data; cyclically reading full data of each day from the start date to the end date from the offline sample data, and processing and generating sample feature data; cyclically reading online data of each day from the start date to the end date from the offline sample data, and processing and generating online feature data; associating offline feature data, sample feature data, and online feature data according to date and sample ID to generate a wide table of training samples.

[0058] In specific fields such as finance, users often provide feedback on recommendation results by replying to text messages or clicking on WeChat push notifications. That is, the online push results on the Nth day are often not fully collected until the N+Mth day (replies within M days are valid, and M is the output time). In order to ensure the consistency of online and offline models, samples need to be divided according to the push time. Therefore, for the push samples of a certain day, a time buffer of M days needs to be reserved. Among them, stdate is the start date, eddate is the end date, middate is the split date of the training set and the validation set, valdays is the number of validation days, and trndays is the number of training days. Training set date [stdate, middate), validation set date [middate, eddate). Among them,

[0059] eddate=todayDate–days(M);

[0060] middate=eddate-days(valdays);

[0061] stdate=eddate-days(trndays);

[0062] The model will be trained using the data in [stdate, middate) and validated using the data in [middate, eddate).

[0063] Figure 4 The flowchart of generating a wide table of training samples in an embodiment of the present invention includes: after the timing task of the current cycle starts, stdate and eddate are obtained, and in one branch, the offline data of lstDate with the date of stdate is read and processed to generate offline feature data; in the other two branches, starting from day=stdate, the data of day i is processed cyclically to become sample data and online feature data until eddate. Finally, the sample and feature table are associated by date and ID to generate a wide table of training samples.

[0064] During automatic model training, the wide table of training samples needs to be split into training set and validation set. The model automatically splits the training set and validation set according to the above stdate, eddate, and middate to facilitate model testing. Figure 5 It is a flowchart of the automatic training model of an embodiment of the present invention, including: after the scheduled task of the current cycle is started, stdate, eddate, and middate are obtained, the training set and the verification set are split according to middate, the training set is used to perform model training, the verification set is used to perform model verification, and finally the model is saved.

[0065] In one implementation of this embodiment, publishing the trained target network model includes: converting the model file of the trained target network model into an ONNX model using the Open Neural Network Exchange (ONNX) format; and publishing the ONNX model.

[0066] Figure 6 This is a flowchart of an embodiment of the present invention for automatically publishing a model, including: after the scheduled task of the current cycle is started, the model file is converted using ONNX, the schema is saved, the ONNX model is published, and online calls are awaited. The model file and the schema file are converted using ONNX, and automatically published for online calls.

[0067] Optionally, after publishing the trained target network model, it also includes: responding to the prediction request of the target network model online, obtaining the online feature data carried in the prediction request; using the online feature data to generate input feature data based on the offline sample data; inputting the input feature data into the target network model, and outputting the prediction result online.

[0068] Figure 7 This is a flowchart of offline training and online publishing of the model in an embodiment of the present invention. The parts that need to be automated in the entire training and publishing process include: feature automation processing (obtaining offline sample data), sample automation processing (generating a wide table of training samples), model automation training, and model automation publishing. The model training and publishing process can be divided into two parts: online and offline.

[0069] In one application scenario, the online process includes:

[0070] Online feature processing: the real-time transaction-related data in the online request is processed into online features, such as the transaction time, transaction type, etc.

[0071] Offline feature query: Get offline features in the offline feature wide table through the ID in the online request;

[0072] Feature splicing: splicing online features with offline features;

[0073] The model predicts the scoring results, and the spliced ​​features are transmitted to the online model deployed in the previous cycle, and the model outputs the scoring results.

[0074] Correspondingly, the offline process includes:

[0075] After the data is displayed on the online front end, user feedback is obtained, and the data is returned and saved in the offline database;

[0076] Label extraction, extracting training labels from the saved offline data;

[0077] Sample sampling: When the gap between positive and negative samples in the recommendation system is too large, samples need to be sampled for model training;

[0078] Offline feature extraction includes: constructing multi-dimensional features from offline data, such as user portraits, material feature tables, etc., and storing them as offline feature wide tables. The offline data contains online feature-related data, and online and offline transaction-related data need to be processed consistently.

[0079] Generate a wide table of training samples. The samples and feature tables are associated through IDs and concatenated and stored as a wide table of training samples.

[0080] Model training and deployment, model verification, after multiple groups of comparative tests and verification of offline models, models with good results are selected for release, model training, the models to be released in the offline models will be automatically trained and released on a regular basis.

[0081] This embodiment can be applied in the model automation training and publishing scenario based on Jupyternotebook, which is a web application that allows users to integrate explanatory text, mathematical equations, codes, and visualization content into an easy-to-share document. The model automation training and publishing solution based on Jupyternotebook can integrate functions such as scheduling documents, model equations, codes, and model test comparison visualization.

[0082] This embodiment can be applied to the automated training and deployment of models, automated feature processing, automated sample processing, automated model training, and automated model publishing in scenarios such as finance where data is sensitive and cannot be migrated. Use python to determine whether an HDFS path exists: Use python's subprocess to call the hadoop command to determine whether a certain path exists. Various data are appended by date in the specified directory of HDFS, and pyspark (an API in the Python environment) is used to splice the root directory according to the specified directory, and then stored, saving a lot of space and time.

[0083] The solution of this embodiment has little dependence on the development environment. Traditional automated scheduling systems often require the interaction of multiple programming languages ​​and multiple platforms, while this solution has simple environmental requirements and only needs to support pyspark, which can read and write HDFS directories. The solution of this embodiment is highly flexible. When the overall framework is determined, the automation of each part can flexibly display relevant information according to the needs of the financial business side, such as model training logs, historical model verification records, etc.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0085] Example 2

[0086] In this embodiment, a publishing device for a network model is also provided to implement the above-mentioned embodiments and preferred implementation modes, which have been described and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0087] Figure 8 is a structural block diagram of a publishing device for a network model according to an embodiment of the present invention, such as Figure 8 As shown, the device includes: an acquisition module 80, a first generation module 82, a splitting module 84, and a publishing module 86, wherein:

[0088] An acquisition module 80 is used to acquire offline sample data of the current period according to the file directory of the target device;

[0089] A first generating module 82 is used to generate a training sample wide table according to the output duration of the offline sample data, wherein the training sample wide table is used to store sample data according to time slices;

[0090] A splitting module 84 is used to split the training sample wide table into a training set and a validation set;

[0091] The publishing module 86 is used to iteratively update the historical network model of the previous cycle using the training set and the verification set on the target device, and publish the trained target network model.

[0092] Optionally, the acquisition module includes: a determination unit, used to determine the last settlement date of the previous period of the current period; a search unit, used to search for a file directory storing the last settlement date in a preset database of the target device; and a generation unit, used to generate offline sample data of the current period based on the file directory.

[0093] Optionally, the search unit includes: a calling subunit, used to use a subprocess to call a hadoop command in the target device; a judging subunit, used to execute the hadoop command and judge whether there is a Hadoop distributed file system HDLS path of the last settlement date in the preset database; and a determining subunit, used to determine the data under the corresponding path as the file directory stored on the last settlement date if the HDLS path of the last settlement date exists in the preset database.

[0094] Optionally, the generation unit includes: an extraction subunit, used to extract feature items from the file directory, wherein each feature item corresponds to a feature dimension of the target sample; and a splicing subunit, used to associate the feature items with feature columns of a feature table through sample IDs, and splice to generate an offline feature wide table of the current cycle.

[0095] Optionally, the first generation module includes: a determination unit, used to determine the end date and start date of the current cycle according to the production duration of the offline sample data; a processing unit, used to read the offline data of the day before the start date from the offline sample data, and process and generate offline feature data; cyclically read the full data of each day from the start date to the end date from the offline sample data, and process and generate sample feature data; cyclically read the online data of each day from the start date to the end date from the offline sample data, and process and generate online feature data; a generation unit, used to associate the offline feature data, the sample feature data, and the online feature data according to the date and sample ID to generate the training sample wide table.

[0096] Optionally, the publishing module includes: a conversion unit, used to convert the model file of the trained target network model into an ONNX model using the open neural network exchange ONNX format; and a publishing unit, used to publish the ONNX model.

[0097] Optionally, the device also includes: a response module, which is used to respond to the prediction request of the target network model online after the publishing module publishes the trained target network model, and obtain the online feature data carried in the prediction request; a second generation module, which is used to use the online feature data to generate input feature data based on the offline sample data; and a prediction module, which is used to input the input feature data into the target network model and output the prediction result online.

[0098] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0099] Example 3

[0100] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0101] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0102] S1, obtain offline sample data of the current period according to the file directory of the target device;

[0103] S2, generating a training sample wide table according to the output duration of the offline sample data, wherein the training sample wide table is used to store sample data according to time slices;

[0104] S3, splitting the training sample wide table into a training set and a validation set;

[0105] S4, iteratively updating the historical network model of the previous cycle using the training set and the verification set on the target device, and publishing the trained target network model.

[0106] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0107] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0108] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0109] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0110] S1, obtain offline sample data of the current period according to the file directory of the target device;

[0111] S2, generating a training sample wide table according to the output duration of the offline sample data, wherein the training sample wide table is used to store sample data according to time slices;

[0112] S3, splitting the training sample wide table into a training set and a validation set;

[0113] S4, iteratively updating the historical network model of the previous cycle using the training set and the verification set on the target device, and publishing the trained target network model.

[0114] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0115] Fig. 9 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Fig. 9 As shown, it includes a processor 91, a communication interface 92, a memory 93 and a communication bus 94, wherein the processor 91, the communication interface 92 and the memory 93 communicate with each other via the communication bus 94, the memory 93 is used to store computer programs, and the processor 91 is used to execute the programs stored in the memory 93.

[0116] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0117] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0122] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for publishing a network model, characterized in that: include: Obtain offline sample data of the current period according to the file directory of the target device; Generate a training sample wide table according to the output duration of the offline sample data, wherein the training sample wide table is used to store sample data according to time slices; Splitting the training sample wide table into a training set and a validation set; Iteratively updating the historical network model of the previous cycle using the training set and the verification set on the target device, and publishing the trained target network model; Among them, generating a training sample wide table according to the output duration of the offline sample data includes: determining the end date and the start date of the current cycle according to the output duration of the offline sample data; reading the offline data of the day before the start date from the offline sample data, and processing and generating offline feature data; cyclically reading the full data of each day from the start date to the end date from the offline sample data, and processing and generating sample feature data; cyclically reading the online data of each day from the start date to the end date from the offline sample data, and processing and generating online feature data; associating the offline feature data, the sample feature data, and the online feature data according to the date and sample ID to generate the training sample wide table; obtaining the training sample wide table according to the file directory of the target device. Taking offline sample data of the current cycle includes: determining the last settlement date of the cycle before the current cycle; searching for the file directory stored on the last settlement date in the preset database of the target device; generating offline sample data of the current cycle according to the file directory; wherein, searching for the file directory stored on the last settlement date in the preset database of the target device includes: using a child process to call the hadoop command in the target device; executing the hadoop command, and determining whether there is a Hadoop distributed file system HDLS path of the last settlement date in the preset database; if the HDLS path of the last settlement date exists in the preset database, determining the data under the corresponding path as the file directory stored on the last settlement date.

2. The method according to claim 1, characterized in that Generating offline sample data of the current period according to the file directory includes: Extracting feature items in the file directory, wherein each feature item corresponds to a feature dimension of a target sample; The feature items are associated with the feature columns of the feature table through the sample ID, and the offline feature wide table of the current cycle is generated by splicing.

3. The method according to claim 1, characterized in that The target network models that have been trained include: Use the Open Neural Network Exchange (ONNX) format to convert the model file of the trained target network model into an ONNX model; Release the ONNX model.

4. The method according to claim 1, characterized in that: After publishing the trained target network model, the method further includes: Responding online to the prediction request of the target network model, and obtaining online feature data carried in the prediction request; Using the online feature data to generate input feature data based on the offline sample data; The input feature data is input into the target network model, and the prediction result is output online.

5. A network model publishing device, characterized in that: include: The acquisition module is used to obtain the offline sample data of the current period according to the file directory of the target device; A first generating module, configured to generate a training sample wide table according to the output duration of the offline sample data, wherein the training sample wide table is used to store the sample data according to time slices; A splitting module, used for splitting the training sample wide table into a training set and a validation set; A publishing module, used to iteratively update the historical network model of the previous cycle on the target device using the training set and the verification set, and publish the trained target network model; The first generation module includes: a determination unit, which is used to determine the end date and the start date of the current cycle according to the output duration of the offline sample data; a processing unit, which is used to read the offline data of the day before the start date from the offline sample data, and process and generate offline feature data; cyclically read the full data of each day from the start date to the end date from the offline sample data, and process and generate sample feature data; cyclically read the online data of each day from the start date to the end date from the offline sample data, and process and generate online feature data; a generation unit, which is used to associate the offline feature data, the sample feature data, and the online feature data according to the date and sample ID to generate the training sample wide table; Wherein, the acquisition module includes: a determination unit, used to determine the last settlement date of the previous cycle of the current cycle; a search unit, used to search for the file directory stored in the preset database of the target device for the last settlement date; a generation unit, used to generate offline sample data of the current cycle according to the file directory; wherein, the search unit includes: a calling subunit, used to use a subprocess to call the hadoop command in the target device; a judgment subunit, used to execute the hadoop command and judge whether there is a Hadoop distributed file system HDLS path of the last settlement date in the preset database; a determination subunit, used to determine the data under the corresponding path as the file directory stored on the last settlement date if the HDLS path of the last settlement date exists in the preset database.

6. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method steps of any one of claims 1 to 4 when running.

7. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface, and the memory communicate with each other via a communication bus; wherein: Memory, used to store computer programs; A processor, configured to execute the method steps of any one of claims 1 to 4 by running a program stored in a memory.

Images