Method and apparatus for generating a script

Abstract

Embodiments of the present disclosure disclose a method and device for generating a script. A specific implementation of the method comprises: obtaining a training script, wherein the training script comprises a sentence for representing feature extraction and a sentence for representing model training; extracting script information of the training script, wherein the script information comprises training dataset information of a training dataset for model training and a feature engineering operator for feature extraction; and generating a feature extraction script according to the sentence where the training dataset information is located and the sentence where the feature engineering operator is located. The implementation realizes automatic generation of a feature extraction script for a test set and a production environment. Furthermore, manual participation in the model deployment process applied to cloud computing can be reduced, and development efficiency is improved.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more particularly to methods and apparatus for generating scripts. Background Technology

[0002] With the rapid development of internet technology, the application of machine learning (ML) technology is becoming more and more widespread.

[0003] When a machine learning model is trained and then deployed for testing or production prediction, it typically relies on two types of files: feature engineering (including data processing) files and model-related files. Model-related files can usually be obtained from existing machine learning libraries, but feature engineering files for the inference phase are often not directly available. For feature engineering files, the usual approach is for technical personnel to manually write feature extraction scripts based on the feature extraction process from samples during model training. Summary of the Invention

[0004] Embodiments of this disclosure provide methods and apparatus for generating scripts.

[0005] In a first aspect, embodiments of this disclosure provide a method for generating a script, the method comprising: obtaining a training script, wherein the training script includes statements for characterizing feature extraction and statements for characterizing model training; extracting script information of the training script, wherein the script information includes training dataset information of a training dataset for model training and feature engineering operators for feature extraction; and generating a feature extraction script based on the statements containing the training dataset information and the statements containing the feature engineering operators.

[0006] In some embodiments, the above-mentioned extraction of script information from the training script includes: determining the target variable and variable relationship from the training script according to the code syntax matching the training script, wherein the target variable is determined based on the training dataset information; dividing the training script into at least one statement group according to a preset knowledge base, wherein the at least one statement group includes the statement containing the training dataset information and the statement containing the feature engineering operator; and determining the functional label of each statement group in the at least one statement group.

[0007] In some embodiments, generating a feature extraction script based on the statements containing the training dataset information and the statements containing the feature engineering operators includes: deleting statement groups matching preset function labels from the training script based on the function labels of each statement group in at least one statement group, and generating a first initial feature extraction script; writing a loading function into the first initial feature extraction script to generate a second initial feature extraction script; and generating a feature extraction script based on the second initial feature extraction script.

[0008] In some embodiments, generating a feature extraction script based on a second initial feature extraction script includes: in response to determining that the training script includes a statement indicating no-information feature engineering, generating a feature extraction script based on the second initial feature extraction script and the statement indicating no-information feature engineering.

[0009] In some embodiments, the above-described extraction of script information for training scripts includes: in response to determining that the training script includes statements indicating information feature engineering, storing the values ​​indicated by the code indicating information feature engineering.

[0010] In some embodiments, generating a feature extraction script based on a second initial feature extraction script includes: in response to determining that the training script includes statements indicating information feature engineering, performing statement transformation on the statements indicating information feature engineering; and generating a feature extraction script based on the second initial feature extraction script and the transformed statements.

[0011] In some embodiments, obtaining the training script as described above includes: in response to receiving a model release request, obtaining the script indicated by the model release request as the training script.

[0012] Secondly, embodiments of this disclosure provide an apparatus for generating a script, the apparatus comprising: an acquisition unit configured to acquire a training script, wherein the training script includes statements for characterizing feature extraction and statements for characterizing model training; an extraction unit configured to extract script information of the training script, wherein the script information includes training dataset information of a training dataset for model training and feature engineering operators for feature extraction; and a generation unit configured to generate a feature extraction script based on the statements containing the training dataset information and the statements containing the feature engineering operators.

[0013] In some embodiments, the extraction unit includes: a first determining module configured to determine target variables and variable relationships from the training script based on code syntax matching the training script, wherein the target variables are determined based on training dataset information; a partitioning module configured to partition the training script into at least one statement group based on a preset knowledge base, wherein the at least one statement group includes a statement containing training dataset information and a statement containing feature engineering operators; and a second determining module configured to determine the functional labels of each statement group in the at least one statement group.

[0014] In some embodiments, the generation unit includes: a first generation module configured to delete statement groups matching preset function tags from the training script based on the function tags of each statement group in at least one statement group, and generate a first initial feature extraction script; a second generation module configured to write a loading function into the first initial feature extraction script, and generate a second initial feature extraction script; and a third generation module configured to generate a feature extraction script based on the second initial feature extraction script.

[0015] In some embodiments, the third generation module is further configured to: generate a feature extraction script based on the second initial feature extraction script and the statement indicating no-information feature engineering in response to determining that the training script includes a statement indicating no-information feature engineering.

[0016] In some embodiments, the extraction unit is further configured to: in response to determining that the training script includes statements indicating information feature engineering, store the values ​​indicated by the code indicating information feature engineering.

[0017] In some embodiments, the third generation module includes: a transformation submodule configured to transform the statement indicating information feature engineering in response to determining that the training script includes a statement indicating information feature engineering; and a generation submodule configured to generate a feature extraction script based on a second initial feature extraction script and the transformed statement.

[0018] In some embodiments, the acquisition unit is further configured to: in response to receiving a model release request, acquire the script indicated by the model release request as a training script.

[0019] Thirdly, embodiments of this disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any implementation of the first aspect.

[0020] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0021] The method and apparatus for generating scripts provided in the embodiments of this disclosure first obtain a training script. The training script includes statements for characterizing feature extraction and statements for characterizing model training. Then, script information of the training script is extracted. This script information includes training dataset information for the training dataset used for model training and feature engineering operators for feature extraction. Finally, a feature extraction script is generated based on the statements containing the training dataset information and the statements containing the feature engineering operators. This achieves automatic generation of feature extraction scripts for test sets and production environments, thereby reducing manual intervention in the model deployment process and improving development efficiency. Attached Figure Description

[0022] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0023] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0024] Figure 2 This is a flowchart of one embodiment of the method for generating scripts according to the present disclosure;

[0025] Figure 3 This is a schematic diagram of an application scenario of a method for generating scripts according to embodiments of the present disclosure;

[0026] Figure 4 This is a flowchart of yet another embodiment of the method for generating scripts according to the present disclosure;

[0027] Figure 5 This is a schematic diagram of one embodiment of the apparatus for generating scripts according to the present disclosure;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0029] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] Figure 1 An exemplary architecture 100 is shown that can be applied to the method or apparatus for generating scripts disclosed herein.

[0032] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0033] Terminal devices 101, 102, and 103 interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, search applications, instant messaging tools, email clients, software development applications, etc.

[0034] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays and supporting model-based software development, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0035] Server 105 can be a server providing various services, such as a backend server supporting the deployment of machine learning models trained on terminal devices 101, 102, and 103. Optionally, server 105 can also be a cloud server. The backend server can analyze and process the received training scripts and generate processing results (such as feature extraction scripts). Optionally, server 105 can further deploy the generated feature extraction scripts and model training scripts to a production environment.

[0036] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0037] It should be noted that the method for generating scripts provided in the embodiments of this disclosure is generally executed by server 105, and correspondingly, the method apparatus for generating scripts is generally located in server 105.

[0038] It should be noted that terminal devices 101, 102, and 103 can also be used to execute the method for generating scripts, and correspondingly, the method device for generating scripts can also be set in terminal devices 101, 102, and 103.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of a method for generating a script according to the present disclosure. The method for generating a script includes the following steps:

[0041] Step 201: Obtain the training script.

[0042] In this embodiment, the execution body of the method used to generate the script (such as...) Figure 1 The server 105 shown can obtain the training script via a wired or wireless connection. The training script may include statements for feature extraction and statements for model training. Typically, the training script is a script written by developers to train a machine learning model. The statements for feature extraction and model training in the training script can respectively constitute the feature extraction code and the model training code. The feature extraction code may be code used to extract features from training samples to generate feature vectors. The model training code may be code used to train an initial model using the feature vectors generated from the training samples and the annotation information of the training samples.

[0043] In some optional implementations of this embodiment, the execution entity may also, in response to receiving a model deployment request, obtain the script indicated by the model deployment request as a training script. For example, the model deployment request may be a request sent by the developer's terminal to the server, indicating that the trained model will be deployed to the production environment. The trained model indicated by the model deployment request may correspond to a script file. The script file may include code for feature extraction and code for model training. In the above approach, the execution entity for generating the script may be the server. As another example, the model deployment request may be a request triggered by the developer clicking the "Deploy" button on the interface of their terminal. In these implementations, the execution entity for generating the script may be the developer's terminal.

[0044] Step 202: Extract the script information of the training script.

[0045] In this embodiment, the execution entity can extract script information for the training script in various ways. This script information may include training dataset information for the training dataset used for model training and feature engineering operators for feature extraction. The training dataset information may include various information characterizing the training dataset, such as storage addresses or variable names, etc.

[0046] In this embodiment, the execution entity can pre-store a feature engineering operator library. The execution entity can parse the training script obtained in step 201 and extract the feature engineering operators included in the training script based on the matching of the feature engineering operator library. These feature engineering operators may include, but are not limited to, at least one of the following: HOG (Histogram of Oriented Gradient), LBP (Local Binary Pattern), Laplace operator, and FAST (Features from Accelerated Segment Test). The execution entity can also extract the training dataset information according to the code writing rules of the training script obtained in step 201. For example, the address after the character "load" can be extracted as the storage address of the training dataset.

[0047] In some optional implementations of this embodiment, the execution entity may further extract the script information of the training script according to the following steps:

[0048] The first step is to determine the target variable and variable relationships from the training script based on the code syntax that matches the training script.

[0049] In these implementations, based on the code syntax matching the training script, the executing entity can determine the target variables and variable relationships from the training script according to the rules of the code syntax. The target variables can be determined based on the training dataset information. For example, the target variables can be variables representing the training data in the training dataset indicated by the training dataset information. The variable relationships can be used to represent various associations between the target variables. These associations can include, but are not limited to, at least one of the following: inheritance, implementation, dependency, and composition. Optionally, these associations can also be represented using Unified Modeling Language (UML) categories. Typically, the executing entity can determine the variable names included in the script based on the syntax of the code writing language of the training script. Thus, the variable relationships between the target variables indicated by the variable names can be further determined. Optionally, the executing entity can also further determine the computational evolution process of the target variables based on the target variables and variable relationships.

[0050] The second step is to divide the training script into at least one statement group based on the preset knowledge base.

[0051] In these implementations, based on a preset knowledge base, the execution entity can further divide the training script obtained in step 201 into at least one statement group. The preset knowledge base can include the correspondence between various statement combinations and functional labels. The statement combinations can include, but are not limited to, at least one of the following: functional modules, algorithm packages, and feature engineering class encapsulation methods. The functional modules can include commonly used statements for characterizing data reading, data loading, and data cleaning. The algorithm packages can include visualization algorithm packages, feature engineering algorithm packages, and modeling algorithm packages. The feature engineering class encapsulation methods can include chi-square binning. It can be understood that the at least one statement group can include the statement containing the training dataset information and the statement containing the feature engineering operator. Therefore, the execution entity can divide the script into different statement groups based on the functional labels corresponding to the statement combinations formed by different statements.

[0052] The third step is to determine the function labels for each statement group within at least one statement group.

[0053] In these implementation methods, based on the statement groups divided in the second step above, the execution entity can determine the function labels corresponding to each statement group in at least one statement group according to the above correspondence.

[0054] Optionally, the aforementioned execution entity may further annotate at least one of the statements, variables, and functions in each statement group with functional labels.

[0055] Step 203: Generate a feature extraction script based on the statements containing the training dataset information and the statements containing the feature engineering operators.

[0056] In this embodiment, based on the statements containing the training dataset information and the feature engineering operators extracted in step 202, the execution entity can generate a feature extraction script in various ways. For example, the execution entity can extract statements including the extracted training dataset information and feature engineering operators and combine them in the same order as in the training script to generate an initial script. Thus, statements for feature extraction of the training dataset information can be extracted from the training script. Then, the execution entity can replace the variables indicated by the training dataset information in the initial script with test variables to generate a new script. Thus, the new script can extract features from the data represented by the test variables using a method consistent with the feature extraction in the training script. It can be understood that the new script can be the feature extraction script.

[0057] In some optional implementations of this embodiment, based on the functional labels of each statement group determined in the optional implementations, the execution entity can generate a feature extraction script according to the following steps:

[0058] The first step is to delete the statement groups that match the preset function labels from the training script based on the function labels of each statement group in at least one statement group, and generate the first initial feature extraction script.

[0059] In these implementations, based on the function labels of each statement group in at least one statement group, the execution entity can delete statement groups matching preset function labels from the training script to generate a first initial feature extraction script. The preset function labels can be function labels unrelated to feature extraction. These preset function labels can include, but are not limited to, at least one of the following: data visualization, model building, and model evaluation.

[0060] The second step is to write the loading function into the first initial feature extraction script to generate the second initial feature extraction script.

[0061] In these implementations, the executing entity can further incorporate a loading function into the first initial feature extraction script generated in the first step, and define the first initial feature extraction script, including the loading function, as the second initial feature extraction script. The loading function can be used to load the target dataset. This allows for the execution of other operations indicated by the code in the script on the target dataset. The target dataset can be related to the application environment of the model corresponding to the training script. For example, if the trained model is deployed to a test environment, the target dataset can be a test dataset. As another example, if the trained model can be deployed to a real production environment, the target dataset can be a dataset used for prediction using the trained model.

[0062] The third step is to generate a feature extraction script based on the second initial feature extraction script.

[0063] In these implementations, the executing entity can directly determine the second initial feature extraction as the feature extraction sample.

[0064] Optionally, in response to determining that the training script includes a statement indicating informationless feature engineering, the execution entity can also generate a feature extraction script based on the second initial feature extraction script and the statement indicating informationless feature engineering. The informationless feature engineering can be used to characterize a transformation process that extracts features without utilizing other information. Examples include exponential transformations, logarithmic transformations, etc. Another example is converting values ​​greater than 0 to 1 and values ​​less than or equal to 0 to -1. Since the information transformation process of informationless feature engineering is independent of the processed data, the original statement can be directly reused. Therefore, the execution entity can determine the second initial feature extraction script, which retains the statement indicating informationless feature engineering, as a feature extraction sample.

[0065] See also Figure 3 , Figure 3 This is a schematic diagram illustrating an application scenario of a method for generating scripts according to embodiments of the present disclosure. Figure 3In this application scenario, developer 301 uses terminal device 302 to write training script 303. Training script 303 includes statements for the machine learning model and statements for feature extraction from training samples. After the machine learning model is trained, developer 301 deploys it via terminal device 302. Terminal device 302 sends training script 303 to server 304. Server 304 first extracts script information 305 from training script 303. Then, based on script information 305, server 304 generates feature extraction script 306. Thus, server 304 can also use the generated feature extraction script 306 to provide feature extraction services for the machine learning model corresponding to training script 303.

[0066] Currently, one common technique involves technicians writing feature extraction scripts after model training, resulting in significant manpower and time costs. However, the method provided in the embodiments of this disclosure, through targeted processing (e.g., statement extraction and replacement) of the feature extraction code within the training script, automatically generates feature extraction scripts for both test sets and production environments. This reduces manual intervention during model deployment, saves time, and improves development efficiency, making it particularly suitable for large-scale development and deployment needs such as cloud computing.

[0067] Further reference Figure 4 This illustrates a flow 400 of yet another embodiment of a method for generating scripts. Flow 400 of this method for generating scripts includes the following steps:

[0068] Step 401: Obtain the training script.

[0069] In some optional implementations of this embodiment, the execution entity may also, in response to receiving a model release request, obtain the script indicated by the model release request as a training script.

[0070] Step 402: Determine the target variable and variable relationships from the training script based on the code syntax that matches the training script.

[0071] Step 403: Based on the preset knowledge base, divide the training script into at least one statement group.

[0072] Step 404: Determine the function labels of each statement group in at least one statement group.

[0073] Step 405: In response to determining that the training script includes statements indicating information feature engineering, the values ​​indicated by the code indicating information feature engineering are stored.

[0074] In this embodiment, the aforementioned informative feature engineering can be used to characterize the transformation process that requires feature extraction using the feature values ​​of the processed data. The transformation process can further include supervised transformation and unsupervised transformation. The feature values ​​of the processed data can include statistical information. This statistical information can include, but is not limited to, at least one of the following: mean, standard deviation, upper and lower bounds, standardization, and dimensionality reduction using principal component analysis (PCA).

[0075] It should be noted that the unsupervised transformation mentioned above usually refers to a transformation that only uses the feature values ​​of the processed data; the supervised transformation mentioned above usually refers to a transformation that uses both the feature values ​​of the processed data and the target value information, such as dimensionality reduction using Linear Discriminant Analysis (LDA) and feature selection using a pre-trained model.

[0076] Step 406: Based on the function labels of each statement group in at least one statement group, delete the statement groups that match the preset function labels from the training script to generate the first initial feature extraction script.

[0077] Step 407: Write the loading function into the first initial feature extraction script to generate the second initial feature extraction script.

[0078] Step 401 can be the same as step 201 in the previous embodiment. Steps 402, 403, and 404 can be the same as optional implementations of step 202 in the previous embodiment. Steps 406 and 407 can be the same as optional implementations of step 203 in the previous embodiment. The descriptions of optional implementations of steps 201, 202, and 203 above also apply to steps 401, 402 to 404, 406, and 407, and will not be repeated here.

[0079] Step 408: Generate a feature extraction script based on the second initial feature extraction script.

[0080] In this embodiment, step 408 can be consistent with the description of the optional implementation of step 203 in the previous embodiment, and will not be repeated here.

[0081] In some optional implementations of this embodiment, the execution entity may also generate a feature extraction script through the following steps:

[0082] The first step is to transform the statements that indicate information feature engineering in response to the determination that the training script includes such statements.

[0083] Based on the aforementioned optional implementation methods, in response to determining that the training script includes statements indicating information feature engineering, the execution entity can typically transform the statements according to the programming language of the code indicating the information feature engineering. As an example, the programming language could be Python. When the statement indicating information feature engineering is a `fit` statement, the execution entity can transform the `fit` statement into a `transform` statement.

[0084] The second step is to generate a feature extraction script based on the second initial feature extraction script and the transformed statements.

[0085] Based on the aforementioned optional implementation methods, the execution entity can generate the feature extraction script in various ways. As an example, the execution entity can transform all statements in the second initial feature extraction script that indicate information feature engineering using the method described above, and determine the transformed script as the feature extraction script.

[0086] from Figure 4 As can be seen from the diagram, the process 400 of the method for generating the script in this embodiment embodies the step of storing the values ​​indicated by the statements indicating information feature engineering. Therefore, the scheme described in this embodiment can provide a data foundation for the transformation of information feature engineering, thereby realizing the generation of feature extraction scripts suitable for the prediction set based on the feature engineering statements in the training script.

[0087] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an apparatus for generating scripts, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0088] like Figure 5 As shown, the apparatus 500 for generating scripts provided in this embodiment includes an acquisition unit 501, an extraction unit 502, and a generation unit 503. The acquisition unit 501 is configured to acquire a training script, which includes statements for characterizing feature extraction and statements for characterizing model training. The extraction unit 502 is configured to extract script information from the training script, which includes training dataset information for model training and feature engineering operators for feature extraction. The generation unit 503 is configured to generate a feature extraction script based on the statements containing the training dataset information and the statements containing the feature engineering operators.

[0089] In this embodiment, the specific processing of the acquisition unit 501, extraction unit 502, and generation unit 503 in the script generation device 500, and the resulting technical effects, can be found in reference to [reference needed]. Figure 2The relevant descriptions of steps 201, 202 and 203 in the corresponding embodiments will not be repeated here.

[0090] In some optional implementations of this embodiment, the extraction unit 502 may include a first determining module (not shown in the figure), a partitioning module (not shown in the figure), and a second determining module (not shown in the figure). The first determining module may be configured to determine the target variable and variable relationships from the training script based on code syntax matching the training script. The target variable may be determined based on training dataset information. The partitioning module may be configured to divide the training script into at least one statement group according to a preset knowledge base. The at least one statement group may include statements containing training dataset information and statements containing feature engineering operators. The second determining module may be configured to determine the functional labels of each statement group within the at least one statement group.

[0091] In some optional implementations of this embodiment, the generation unit 503 may include a first generation module (not shown in the figure), a second generation module (not shown in the figure), and a third generation module (not shown in the figure). The first generation module may be configured to delete statement groups matching preset function tags from the training script based on the function tags of each statement group in at least one statement group, thereby generating a first initial feature extraction script. The second generation module may be configured to write a loading function into the first initial feature extraction script, thereby generating a second initial feature extraction script. The third generation module may be configured to generate a feature extraction script based on the second initial feature extraction script.

[0092] In some optional implementations of this embodiment, the third generation module may be further configured to: generate a feature extraction script based on the second initial feature extraction script and the statement indicating no-information feature engineering in response to determining that the training script includes a statement indicating no-information feature engineering.

[0093] In some optional implementations of this embodiment, the extraction unit 502 may be further configured to: in response to determining that the training script includes a statement indicating information feature engineering, store the value indicated by the code indicating information feature engineering.

[0094] In some optional implementations of this embodiment, the third generation module may include a transformation submodule (not shown in the figure) and a generation submodule (not shown in the figure). The transformation submodule may be configured to transform the statements indicating information feature engineering in response to determining that the training script includes such statements. The generation submodule may be configured to generate a feature extraction script based on the second initial feature extraction script and the transformed statements.

[0095] In some optional implementations of this embodiment, the acquisition unit 501 can be further configured to: in response to receiving a model release request, acquire the script indicated by the model release request as a training script.

[0096] The apparatus provided in the above embodiments of this disclosure first acquires a training script through an acquisition unit 501. The training script includes statements for characterizing feature extraction and statements for characterizing model training. Then, an extraction unit 502 extracts script information from the training script. This script information includes training dataset information for the training dataset used for model training and feature engineering operators for feature extraction. Finally, a generation unit 503 generates a feature extraction script based on the statements containing the training dataset information and the statements containing the feature engineering operators. This achieves automatic generation of feature extraction scripts for both test sets and production environments. Consequently, it reduces manual intervention in the model deployment process and improves development efficiency.

[0097] The following is for reference. Figure 6 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 1 The diagram shows the structure of the server (600) in this disclosure. The terminal devices in the embodiments of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, and tablets, as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The server shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0098] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0099] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0100] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by a processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0101] It should be noted that the computer-readable medium described in the embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0102] The aforementioned computer-readable medium may be included within the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a training script, wherein the training script includes statements for characterizing feature extraction and statements for characterizing model training; extract script information from the training script, wherein the script information includes training dataset information for the training dataset used for model training and feature engineering operators for feature extraction; and generate a feature extraction script based on the statements containing the training dataset information and the statements containing the feature engineering operators.

[0103] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] The units described in the embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, it can be described as: a processor including an acquisition unit, an extraction unit, and a generation unit. The names of these units do not necessarily limit the unit itself; for example, the acquisition unit can also be described as "a unit that acquires training scripts, wherein the training scripts include statements for characterizing feature extraction and statements for characterizing model training."

[0106] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating a script, comprising: obtaining a training script, wherein the training script comprises a statement for representing feature extraction and a statement for representing model training; extracting script information of the training script, comprising: traversing and parsing the training script, and extracting a feature engineering operator included in the training script for the feature extraction according to matching of a pre-stored feature engineering operator library; and extracting training dataset information of a training dataset for the model training according to a code writing rule of the training script; generating a feature extraction script according to a statement where the training dataset information is located and a statement where the feature engineering operator is located.

2. The method of claim 1, wherein, The extracting of the script information of the training script comprises: determining a target variable and a variable relationship from the training script according to a code syntax matched with the training script, wherein the target variable is determined based on the training dataset information; dividing the training script into at least one statement group according to a pre-set knowledge base, wherein the at least one statement group comprises the statement where the training dataset information is located and the statement where the feature engineering operator is located; determining a function label of each statement group in the at least one statement group.

3. The method of claim 2, wherein, The generating of the feature extraction script according to the statement where the training dataset information is located and the statement where the feature engineering operator is located comprises: generating a first initial feature extraction script by deleting, from the training script, a statement group matched with a pre-set function label according to the function label of each statement group in the at least one statement group; writing a loading function into the first initial feature extraction script to generate a second initial feature extraction script; generating the feature extraction script based on the second initial feature extraction script.

4. The method of claim 3, wherein, The generating of the feature extraction script based on the second initial feature extraction script comprises: in response to determining that the training script comprises a statement indicating information-free feature engineering, generating the feature extraction script based on the second initial feature extraction script and the statement indicating the information-free feature engineering.

5. The method of claim 3, wherein, The extracting of the script information of the training script comprises: in response to determining that the training script comprises a statement indicating information-rich feature engineering, storing a value indicated by the code of the statement indicating the information-rich feature engineering.

6. The method of claim 5, wherein, The generating of the feature extraction script based on the second initial feature extraction script comprises: in response to determining that the training script comprises a statement indicating information-rich feature engineering, performing statement transformation on the statement indicating the information-rich feature engineering; generating the feature extraction script based on the second initial feature extraction script and the transformed statement.

7. The method according to one of claims 1 to 6, wherein The obtaining of the training script comprises: in response to receiving a model publishing request, obtaining a script indicated by the model publishing request as the training script.

8. An apparatus for generating a script, comprising: an obtaining unit configured to obtain a training script, wherein the training script comprises a statement for representing feature extraction and a statement for representing model training; The extraction unit is configured to extract script information of the training script, including: traversing and parsing the training script, and extracting a feature engineering operator included in the training script for feature extraction according to matching of a pre-stored feature engineering operator library; and extracting training data set information of a training data set for the model training according to a code writing rule of the training script; The generation unit is configured to generate a feature extraction script according to a sentence where the training data set information is located and a sentence where the feature engineering operator is located.

9. The apparatus of claim 8, wherein, The extraction unit includes: A first determination module configured to determine a target variable and a variable relationship from the training script according to a code syntax matched with the training script, wherein the target variable is determined based on the training data set information; A division module configured to divide the training script into at least one sentence group according to a pre-set knowledge base, wherein the at least one sentence group includes the sentence where the training data set information is located and the sentence where the feature engineering operator is located; A second determination module configured to determine a function label of each sentence group in the at least one sentence group.

10. The apparatus of claim 9, wherein, The generation unit includes: A first generation module configured to delete a sentence group matched with a pre-set function label from the training script according to the function label of each sentence group in the at least one sentence group, and generate a first initial feature extraction script; A second generation module configured to write a loading function into the first initial feature extraction script, and generate a second initial feature extraction script; A third generation module configured to generate the feature extraction script based on the second initial feature extraction script.

11. The apparatus of claim 10, wherein, The third generation module is further configured to: In response to determining that the training script includes a sentence indicating information-free feature engineering, generate the feature extraction script based on the second initial feature extraction script and the sentence indicating information-free feature engineering.

12. The apparatus of claim 10, wherein, The extraction unit is further configured to: In response to determining that the training script includes a sentence indicating information-rich feature engineering, store a value indicated by the code indicating information-rich feature engineering.

13. The apparatus of claim 12, wherein, The third generation module includes: A transformation submodule configured to perform sentence transformation on the sentence indicating information-rich feature engineering in response to determining that the training script includes the sentence indicating information-rich feature engineering; A generation submodule configured to generate the feature extraction script based on the second initial feature extraction script and the transformed sentence.

14. The apparatus of one of claims 8-13, wherein, The acquisition unit is further configured to: In response to receiving a model publishing request, acquire a script indicated by the model publishing request as the training script.

15. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.

16. A computer readable medium having stored thereon a computer program, wherein, The program is executed by the processor to implement the method of any one of claims 1-7. The program is executed by the processor to implement the method of any one of claims 1-7.

Images