Method and device, electronic equipment and storage medium

Abstract

The invention provides a data job program creation method and device, electronic equipment and a storage medium. The method comprises the following steps: starting and running a universal framework program; when the framework program runs, configuration information which is input by a user and aims at a target job program is obtained, and the configuration information comprises job program basic information, a field generation rule and a data filtering rule; and initializing the target operation program according to the configuration information, reading and storing data corresponding to the target operation program, and submitting and executing the target operation program. 'one-time deployment and multi-job sharing 'are realized, and repeated packaging and deployment for each job are avoided; resource overhead is reduced and utilization rate is increased.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data job program creation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of big data processing technology, real-time data stream processing plays an increasingly important role in enterprise applications. The open-source distributed big data processing engine (Apache Flink), as a high-performance, low-latency distributed stream processing framework, has been widely used in scenarios such as real-time computing, event-driven processing, and real-time data warehouse construction. In actual project development, real-time development tasks involving Flink often face a large number of similar data processing requirements with slight differences in details.

[0003] For similar needs, the existing technical solution is to write a separate set of Flink real-time job code for each specific requirement. This means developers need to manually write complete modules for data reading, transformation logic, external service calls, and result output based on business requirements, and then run the job through compilation, packaging, and deployment.

[0004] While the above methods can meet the functional requirements of specific scenarios, they suffer from problems such as serious duplication of development and low efficiency when facing high-frequency, batch, and highly similar job requirements. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a data job creation method, apparatus, electronic device, and storage medium to improve the efficiency of data job creation.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a method for creating a data job program, the method comprising: Start and run the general framework program; When the framework program runs, it obtains configuration information for the target job program input by the user. The configuration information includes: basic information of the job program, field generation rules, and data filtering rules. The target job program is initialized according to the configuration information, the data corresponding to the target job program is read and stored, and the target job program is submitted and executed.

[0007] Optionally, the basic information of the job procedure is stored in a job basic information table, which includes: job procedure name, job procedure type, data source information, and result table information; The field generation rules are stored in the field generation rule table, and the field generation rules include: the fields involved in the target job program and the methods for obtaining the fields; The data filtering rules are stored in the data filtering rule table.

[0008] Optionally, the framework program includes at least: a job initiation class, a field generation method class, and a data filtering class; The steps of initializing the target job program according to the configuration information, reading and storing the data corresponding to the target job program, and submitting and executing the target job program include: Load and run an object instance of the job startup class. When the object instance of the job startup class runs, it initializes the target job program and generates an input data transmission object based on the job program name, job program type, data source information and result table information in the job program basic information table. The input data transmission object is used to store input data. Load and run an object instance of the field generation method class. When the object instance of the field generation method class runs, it generates an output data transmission object based on the field generation rules in the field generation rule table and the input data transmission object, and the output data transmission object is used to store the original output data. Load and run an object instance of the data filtering class. When the object instance of the data filtering class runs, it filters the data in the output data object according to the data filtering rules in the data filtering rule table, and outputs the target data of the target job program according to the result table information. Submit and execute the initialized target job program and the target data.

[0009] Optionally, the step of initializing the target job program and generating an input data transmission object based on the job program name, job program type, data source information, and result table information in the job program basic information table includes: The target job program is initialized based on the job program name, job program type, data source information, and result table information in the job program basic information table. Based on the data source information, determine the input data type; Based on the input data type, an input data transmission object is generated, and the input data indicated by the data source information is stored in the input data transmission object.

[0010] Optionally, the step of generating an output data transmission object based on the field generation rules in the field generation rule table and the input data transmission object includes: According to the acquisition method in the field generation rule, obtain the field values ​​of each field in the field generation rule from the input data transmission object or related source; The output data transmission object is generated based on the field values ​​of each field.

[0011] Optionally, obtaining the field values ​​of each field in the field generation rule from the input data transmission object or a related source according to the acquisition method in the field generation rule includes: If the acquisition method is to obtain from the data source, then the field values ​​of each field in the field generation rule are obtained from the input data transmission object; If the acquisition method is through API, then the API identifier is obtained from the field generation rule table, the target API is initialized according to the API identifier, and the target API is called with each field in the field generation rule as input parameters to obtain the return result of the target API. The value in the return result is used as the field value of each field in the field generation rule.

[0012] Optionally, the step of filtering the data in the output data object according to the data filtering rules in the data filtering rule table, and outputting the target data of the target job program according to the result table information, includes: The data in the output data object is filtered according to the data filtering rules in the data filtering rule table to obtain filtered data; Determine the data output type based on the information in the result table; The data output class corresponding to the data output type is called to output the filtered data according to the data output type, thereby obtaining the target data.

[0013] Secondly, embodiments of this application also provide a data job creation apparatus, the apparatus comprising: The startup module is used to start and run general framework programs; The acquisition module is used to acquire configuration information for the target job program input by the user during the runtime of the framework program. The configuration information includes: basic information of the job program, field generation rules, and data filtering rules. The submission module is used to initialize the target job program according to the configuration information, read and store the data corresponding to the target job program, and submit and execute the target job program.

[0014] Optionally, the basic information of the job procedure is stored in a job basic information table, which includes: job procedure name, job procedure type, data source information, and result table information; The field generation rules are stored in the field generation rule table, and the field generation rules include: the fields involved in the target job program and the methods for obtaining the fields; The data filtering rules are stored in the data filtering rule table.

[0015] Optionally, the framework program includes at least: a job initiation class, a field generation method class, and a data filtering class; The submission module is specifically used for: Load and run an object instance of the job startup class. When the object instance of the job startup class runs, it initializes the target job program and generates an input data transmission object based on the job program name, job program type, data source information and result table information in the job program basic information table. The input data transmission object is used to store input data. Load and run an object instance of the field generation method class. When the object instance of the field generation method class runs, it generates an output data transmission object based on the field generation rules in the field generation rule table and the input data transmission object, and the output data transmission object is used to store the original output data. Load and run an object instance of the data filtering class. When the object instance of the data filtering class runs, it filters the data in the output data object according to the data filtering rules in the data filtering rule table, and outputs the target data of the target job program according to the result table information. Submit and execute the initialized target job program and the target data.

[0016] Optionally, the submission module is specifically used for: The target job program is initialized based on the job program name, job program type, data source information, and result table information in the job program basic information table. Based on the data source information, determine the input data type; Based on the input data type, an input data transmission object is generated, and the input data indicated by the data source information is stored in the input data transmission object.

[0017] Optionally, the submission module is specifically used for: According to the acquisition method in the field generation rule, obtain the field values ​​of each field in the field generation rule from the input data transmission object or related source; The output data transmission object is generated based on the field values ​​of each field.

[0018] Optionally, the submission module is specifically used for: If the acquisition method is to obtain from the data source, then the field values ​​of each field in the field generation rule are obtained from the input data transmission object; If the acquisition method is through API, then the API identifier is obtained from the field generation rule table, the target API is initialized according to the API identifier, and the target API is called with each field in the field generation rule as input parameters to obtain the return result of the target API. The value in the return result is used as the field value of each field in the field generation rule.

[0019] Optionally, the submission module is specifically used for: The data in the output data object is filtered according to the data filtering rules in the data filtering rule table to obtain filtered data; Determine the data output type based on the information in the result table; The data output class corresponding to the data output type is called to output the filtered data according to the data output type, thereby obtaining the target data.

[0020] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the application runs, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the data job program creation method described in the first aspect above.

[0021] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which is read and executes the steps of the data job program creation method described in the first aspect.

[0022] The beneficial effects of this application are: This application provides a data job program creation method, apparatus, electronic device, and storage medium. By constructing a resident and running general framework program, it achieves "deploy once, share multiple jobs," avoiding repeated packaging and deployment for each job; reducing resource consumption and improving system utilization; forming a stable and reliable operating environment base; acquiring user-input configuration information at runtime; initializing the target job program based on the configuration information; reading and storing the data corresponding to the target job program; and submitting and executing the target job program. It can separate the code program from the configuration information, allowing non-technical personnel to complete job definition through the interface. Only configuration information is needed to complete the executable target job program. It has strong scalability, significantly improves development efficiency, and avoids repetitive development in the prior art. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram of the architecture of a framework program provided for an embodiment of this application; Figure 2 A flowchart illustrating a data job creation method provided in an embodiment of this application; Figure 3 A flowchart illustrating another data job creation method provided in this application embodiment; Figure 4 A flowchart illustrating a method for generating an input data transmission object provided in an embodiment of this application; Figure 5 A flowchart illustrating a method for outputting target data provided in an embodiment of this application; Figure 6 A complete process diagram for creating a data job program is provided as an embodiment of this application; Figure 7 A schematic diagram of an apparatus for a data job program creation method provided in an embodiment of this application; Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0026] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0027] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0028] Figure 1 A schematic diagram of the architecture of a framework program provided in this application embodiment is shown below. Figure 1 As shown, the framework program may include: a job initiation class, a field generation method class, and a data filtering class.

[0029] The job initiation class is responsible for initializing the target job program and preparing input data, serving as the entry point for the entire job execution process. The field generation method class converts input data into intermediate output data containing new field structures according to preset field generation rules, and is the core processing unit of the data transformation layer. The data filtering class performs conditional filtering on the intermediate data after field generation and finally outputs the target data in a specified format, representing the final processing step before data output.

[0030] The target job program is created using the job startup class, field generation method class, and data filtering class in the framework program, and by utilizing the method steps provided in the embodiments of this application.

[0031] Optionally, the data job creation method provided in this application embodiment can be applied to an electronic device, such as a mobile phone, tablet computer, laptop computer, PDA, desktop computer, or other terminal device with computing and display capabilities, or it can be a server. Specifically, it can be applied to applications in terminal devices, such as mobile phone applications (APP) and computer application systems.

[0032] The following section will explain in detail the specific implementation process of creating the data job program provided in the embodiments of this application.

[0033] Figure 2 This is a flowchart illustrating a data job creation method provided in an embodiment of this application. The execution entity of this method is, as described above, an electronic device with a framework program deployed on it. Figure 2As shown, the method includes: S101. Start and run the general framework program.

[0034] Optionally, this general-purpose framework program can be deployed as a standalone application service on electronic devices. Physically, the general-purpose framework program is a packaged JAR file. When a user triggers a start command, such as submitting a task by calling the "flink run" command via the command line, or clicking the "Start Framework" button in the web console, the general-purpose framework program can be started and run.

[0035] The framework program can be deployed once and reused for multiple tasks, without needing to be recompiled or packaged for each specific job. After starting, this general-purpose framework program does not immediately execute a specific data processing task, but instead enters a standby running state, continuously listening for configuration input events from external systems. Using this general-purpose framework program, multiple different jobs of the same type can be served.

[0036] S102. When the framework program is running, it obtains the configuration information for the target job program input by the user.

[0037] This configuration information may include: basic information about the job program, field generation rules, and data filtering rules.

[0038] Optionally, the configuration information of the target job program can be input by the user through an external interactive interface or system interface. Instead of being fixed in the code during program compilation, it is dynamically loaded during the runtime of the framework program, thereby supporting on-demand generation and real-time execution of jobs.

[0039] Optionally, the configuration information for each job can be pre-written into the framework program. When a particular job needs to be executed, its configuration information can be directly obtained. For example, an e-commerce platform needs to generate multiple product recommendation data stream jobs daily, such as job A and job B. Both job A and job B are jobs that generate multiple product recommendation data streams, but the specific data generated is different. Therefore, for job A, the configuration information for job A can be obtained; for job B, the configuration information for job B can be obtained.

[0040] S103. Initialize the target job program according to the configuration information, read and store the data corresponding to the target job program, and submit and execute the target job program.

[0041] Specifically, after obtaining the configuration information of the target job program, the target job program can be initialized, processed, and stored according to the basic information of the job program, field generation rules, and data filtering rules in the configuration information. A complete and executable target job program can then be constructed, and the constructed target job program can be submitted and executed.

[0042] In this embodiment, a general framework program is started and run. During runtime, the framework program obtains configuration information for the target job program input by the user. This configuration information includes basic job program information, field generation rules, and data filtering rules. Based on the configuration information, the framework program initializes the target job program, reads and stores the corresponding data, and submits and executes the target job program. By constructing a persistent, general framework program, "deploy once, share with multiple jobs" is achieved, avoiding repeated packaging and deployment for each job. This reduces resource overhead, improves system utilization, and forms a stable and reliable operating environment. By obtaining user-input configuration information during runtime, initializing the target job program, reading and storing the corresponding data, and submitting and executing the target job program, the framework program separates the code from the configuration information. This allows non-technical personnel to define jobs through the interface, requiring only the acquisition of configuration information to complete the executable target job program. It offers strong scalability, significantly improves development efficiency, and avoids repetitive development in existing technologies.

[0043] Optionally, the basic information of the aforementioned job procedure is stored in a job basic information table. This basic information includes: job procedure name, job procedure type, data source information, and result table information. The data source information may include the method of obtaining the data source, and the result table information may include the method of outputting the results.

[0044] The job name can be represented by, for example, `jobName`. The job name is used to uniquely identify the job, facilitating scheduling, monitoring, and log tracking. The job type can be represented by `jobType`. The job type indicates the processing mode of the job, such as "real-time stream processing," "batch processing," or "event-driven task," which determines the running parameters of the subsequent execution engine. The data source information can be represented by, for example, `dataSourceInfo`. It can describe the source of the input data, including but not limited to the data source type connection address, authentication credentials, subject / table name, and read mode. The result table information can be represented by, for example, `resultTableInfo`. The result table information can define the target storage location and format of the output data, such as the target database name, table name, write mode, and partitioning strategy.

[0045] Optionally, the field generation rules mentioned above are stored in a field generation rule table. The field generation rules include: the fields involved in the target job program and the methods for obtaining the fields.

[0046] Specifically, the field generation rule table can be a structured configuration storage unit, such as a database table, a JSON configuration file, or a memory cache object. The field generation rule table can include the names of all fields involved in each job program and the method of obtaining each field. The field name identifies the field to be generated or calculated. The field acquisition method indicates the source and generation logic of the field, such as "extracted from input data," "obtained via API call," "calculated based on expression," or "constant assignment," etc.

[0047] Optionally, the aforementioned data filtering rules are stored in a data filtering rule table, which includes the data filtering rules for the target job program.

[0048] Optionally, the above framework program includes at least: a job initiation class, a field generation method class, and a data filtering class.

[0049] Figure 3 A flowchart illustrating another data job creation method provided in this application embodiment is shown below. Figure 3 As shown, the steps in S103 above, including initializing the target job program based on the configuration information, reading and storing the data corresponding to the target job program, and submitting and executing the target job program, may include: S201. Load and run an instance of the job startup class. When the instance of the job startup class runs, it initializes the target job program and generates an input data transmission object based on the job program name, job program type, data source information and result table information in the job program basic information table.

[0050] The input data transfer object is used to store input data.

[0051] Specifically, when the general framework program receives configuration information for a specific target job from the user, it initiates the job initialization process. The first step of this process is to load and run an instance of the job startup class. This job startup class acts as the entry controller for the entire target job execution process, responsible for parsing the basic job information and completing the initial construction and context preparation of the target job. The job startup class completes the initialization of the target job and generates an input data transmission object with the actual data source, providing the input foundation for subsequent data transformation and filtering stages. This achieves decoupling between job configuration and code logic; that is, the same set of job startup class code can adapt to any compliant job basic information, supporting the rapid deployment of new jobs without recompilation.

[0052] S202. Load and run an object instance of the field generation method class. When the object instance of the field generation method class runs, it generates an output data transmission object based on the field generation rules in the field generation rule table and the input data transmission object. The output data transmission object is used to store the original output data.

[0053] The output data transfer object is used to store the raw output data.

[0054] Optionally, after initializing the target job program and constructing the input data transmission object, the second stage of the data processing flow, namely field generation, is entered. This stage is executed by an object instance of the field generation method class. At runtime, the object instance of the field generation method class can generate the output data transmission object based on the field generation rules corresponding to the target job program in the field generation table and using a preset method based on the input data transmission object generated in step S201.

[0055] S203. Load and run an instance of the data filtering class. When the instance of the data filtering class runs, it filters the data in the output data object according to the data filtering rules in the data filtering rule table, and outputs the target data of the target job program according to the result table information.

[0056] Optionally, after generating the output data transmission object, an object instance of the data filtering class can be used to filter the data contained in the output data transmission object generated in step S202 according to the data filtering rules corresponding to the target job program in the data filtering rule table. Combined with the result table information, the data that meets the conditions can be converted into the target format and output to form the target data of the target job program.

[0057] S204. Submit and execute the initialized target job program and target data.

[0058] Optionally, the initialized target job program includes information such as job identifier and job type, and the initialized target job program and the obtained target data are submitted and executed.

[0059] In this embodiment, because the job initiation class, field generation method class, and data filtering class are encapsulated within a general framework program, their functions are highly reusable. This allows multiple target job programs of different types but similar structures to share the instantiation logic of this class, significantly reducing system resource overhead and improving job deployment efficiency. Abstracting the field generation logic into configurable rules and having them uniformly scheduled and executed by the field generation method class achieves a high degree of automation and low-code simplicity in the data transformation process. Compared to traditional hard-coding methods, this significantly reduces the amount of repetitive development work caused by changes in field requirements, making it particularly suitable for enterprise-level data integration scenarios that require frequent adjustments to output fields. By fully configuring the data filtering logic and combining it with a dynamically loaded output adapter mechanism, this achieves high flexibility and low-code deployment capabilities in the data output stage. Compared to the traditional approach of manually writing filtering statements or hard-coding logic, this significantly improves development efficiency and ease of maintenance, making it particularly suitable for enterprise-level real-time data pipeline scenarios that require frequent changes to output strategies or multi-target distribution.

[0060] Figure 4 This application provides a flowchart illustrating a method for generating an input data transmission object, as shown in the embodiments below. Figure 4 As shown, in step S201 above, initializing the target job program and generating an input data transmission object based on the job program name, job program type, data source information, and result table information in the job program basic information table may include: S301. Initialize the target job program based on the job program name, job program type, data source information, and result table information in the job program basic information table.

[0061] Specifically, a separate job instance can be created based on the job program name and job program type, and this job instance can be created within the general framework program. This job instance includes the job program name, job program type, data source information, and result table information.

[0062] The initialization process does not involve code compilation or packaging. Instead, it builds the job structure through dynamic configuration loading, achieving a lightweight startup mechanism that allows for one-time deployment and multiple job reuse.

[0063] S302. Determine the input data type based on the data source information.

[0064] The data source information may include the input data and its format. Specifically, the data source information can be parsed to obtain the input data and its format, thereby determining the input data type.

[0065] For example, if the data source is Kafka and the serialization format is JavaScript Object Notation (JSON), then the input data type is "JSON Structured Message Stream"; if the data source is a relational database (MySQL) table, then the input data type is "Relational Recordset".

[0066] S303. Based on the input data type, generate an input data transmission object and store the input data indicated by the data source information into the input data transmission object.

[0067] Specifically, based on the input data type determined in step S302, an input data transfer object of that type can be dynamically created in memory. This input data transfer object can serve as a temporary data container for encapsulating input data read from an external system.

[0068] In this embodiment, a highly flexible and low-code job creation mechanism is achieved by relying entirely on external configuration tables for job initialization and input data preparation instead of hard-coded logic. Developers do not need to rewrite data access code for each new requirement; they can quickly start a new data processing task simply by adding a configuration record to the job program's basic information table. Furthermore, since the identification of input data types and the generation of transmission objects are automated processes, it possesses good versatility and scalability, adapting to various heterogeneous data sources and significantly reducing the technical barriers and operational costs when building large-scale real-time data pipelines.

[0069] Optionally, the process in S202 above, which filters the data in the output data object according to the data filtering rules in the data filtering rule table and outputs the target data of the target job program according to the result table information, may include: Specifically, according to the acquisition method in the field generation rules, the field values ​​of each field in the field generation rules are obtained from the input data transmission object or the relevant source; and the output data transmission object is generated based on the field values ​​of each field.

[0070] The methods for obtaining fields can include the following: 0: empty - no processing, 1: directly from the data source, 2: obtained based on judgment conditions, 3: obtained through the Application Programming Interface (API), 4: obtained by converting enumeration values, 5: obtained based on QL expressions, and 6: default value.

[0071] The acquired fields and their values ​​are sequentially filled into the output data transmission object to form a complete and semantically rich intermediate data record.

[0072] Optionally, obtaining the field values ​​of each field in the field generation rule from the input data transmission object or related source according to the acquisition method in the field generation rule may include: If the retrieval method is from a data source, then the field values ​​of each field in the field generation rules are retrieved from the input data transfer object. Specifically, you can first retrieve each field in the field generation rules from the data source in the input data transfer object, and simultaneously retrieve the field values ​​from the data source in the input data transfer object.

[0073] If the acquisition method is via API, then the API identifier is obtained from the field generation rule table, the target API is initialized according to the API identifier, and the target API is called with each field in the field generation rule as input parameters. The return result of the target API is obtained, and the value in the return result is used as the field value of each field in the field generation rule.

[0074] Specifically, for the target field being processed, its "Field Acquisition Method" configuration item is read. If the method is "3: Obtain via API", it is determined that the target field needs to be obtained through an external interface call, and the API call process begins. A preset API identifier can be extracted from the API acquisition method. This API identifier is a unique string used to associate with a registered external service interface definition. Based on this API identifier, the corresponding service metadata can be found in the pre-registered API metadata registry. This metadata includes at least: service address, request protocol type, request method, content type, authentication method, and request header template. A lightweight target API is constructed based on this service metadata, completing connection preparation and security authentication. The target field is then used as an input parameter to populate the API request, and the request is sent to the target API. When the target API receives the request, it returns a response result, which is then used as the field value of the target field.

[0075] Figure 5 A flowchart illustrating a method for outputting target data provided in an embodiment of this application is shown below. Figure 5 As shown, in step S203 above, filtering the data in the output data object according to the data filtering rules in the data filtering rule table and outputting the target data of the target job program according to the result table information may include: S401. Filter the data in the output data object according to the data filtering rules in the data filtering rule table to obtain the filtered data.

[0076] Optionally, the data filtering rules for the target job program in the data filtering rule table may include information such as: filtering condition expression, operation type, priority, and activation status. The filtering expression can be a logical judgment condition described using standard syntax; the operation type indicates whether the rule is used to retain or exclude matches; the priority can be used to sort the execution order when multiple conditions coexist; and the activation status controls whether the rule is effective, supporting dynamic start and stop at runtime.

[0077] Optionally, the conditions in the data filtering rules can be loaded in order of priority and applied sequentially to each piece of output data in the output data transmission object.

[0078] S402. Determine the data output type based on the information in the result table.

[0079] Specifically, the result table information of the target job program can be parsed to obtain the data output type in the result table information. The data output type can include table type, data queue type, etc. The data output type in the target job program can include one output type or multiple output types.

[0080] S403. Call the data output class corresponding to the data output type to output the filtered data according to the data output type and obtain the target data.

[0081] Optionally, each data output type corresponds to a data output class. If the result table information contains only one data output type, then the data output class for that data output type can be called. If the result table information contains two data output types, such as table type and data queue type, then the data output classes corresponding to both data output types can be called simultaneously. The filtered data is then output through the called data output classes to obtain the target data.

[0082] Figure 6 This is a complete process diagram of creating a data job program according to an embodiment of this application.

[0083] Figure 7 A schematic diagram of an apparatus for creating a data job program provided in an embodiment of this application is shown below. Figure 7 As shown, the device includes: Startup module 501 is used to start and run general framework programs; The acquisition module 502 is used to acquire configuration information for the target job program input by the user during the runtime of the framework program. The configuration information includes: basic information of the job program, field generation rules, and data filtering rules. The submission module 503 is used to initialize the target job program according to the configuration information, read and store the data corresponding to the target job program, and submit and execute the target job program.

[0084] Optionally, the basic information of the job procedure is stored in a job basic information table, which includes: job procedure name, job procedure type, data source information, and result table information; The field generation rules are stored in the field generation rule table, and the field generation rules include: the fields involved in the target job program and the methods for obtaining the fields; The data filtering rules are stored in the data filtering rule table.

[0085] Optionally, the framework program includes at least: a job initiation class, a field generation method class, and a data filtering class; Optionally, the submission module 503 is specifically used for: Load and run an object instance of the job startup class. When the object instance of the job startup class runs, it initializes the target job program and generates an input data transmission object based on the job program name, job program type, data source information and result table information in the job program basic information table. The input data transmission object is used to store input data. Load and run an object instance of the field generation method class. When the object instance of the field generation method class runs, it generates an output data transmission object based on the field generation rules in the field generation rule table and the input data transmission object, and the output data transmission object is used to store the original output data. Load and run an object instance of the data filtering class. When the object instance of the data filtering class runs, it filters the data in the output data object according to the data filtering rules in the data filtering rule table, and outputs the target data of the target job program according to the result table information. Submit and execute the initialized target job program and the target data.

[0086] Optionally, the submission module 503 is specifically used for: The target job program is initialized based on the job program name, job program type, data source information, and result table information in the job program basic information table. Based on the data source information, determine the input data type; Based on the input data type, an input data transmission object is generated, and the input data indicated by the data source information is stored in the input data transmission object.

[0087] Optionally, the submission module 503 is specifically used for: According to the acquisition method in the field generation rule, obtain the field values ​​of each field in the field generation rule from the input data transmission object or related source; The output data transmission object is generated based on the field values ​​of each field.

[0088] Optionally, the submission module 503 is specifically used for: If the acquisition method is to obtain from the data source, then the field values ​​of each field in the field generation rule are obtained from the input data transmission object; If the acquisition method is through API, then the API identifier is obtained from the field generation rule table, the target API is initialized according to the API identifier, and the target API is called with each field in the field generation rule as input parameters to obtain the return result of the target API. The value in the return result is used as the field value of each field in the field generation rule.

[0089] Optionally, the submission module 503 is specifically used for: The data in the output data object is filtered according to the data filtering rules in the data filtering rule table to obtain filtered data; Determine the data output type based on the information in the result table; The data output class corresponding to the data output type is called to output the filtered data according to the data output type, thereby obtaining the target data.

[0090] Figure 8 A structural frame of an electronic device 600 provided in this application embodiment Figure 8 .like Figure 8 As shown, the electronic device may include: a processor 601 and a memory 602.

[0091] Optionally, a bus 603 may also be included, wherein the memory 602 is used to store machine-readable instructions executable by the processor 601. When the electronic device 600 is running, the processor 601 and the memory 602 communicate via the bus 603. When the machine-readable instructions are executed by the processor 601, the method steps in the above method embodiments are performed.

[0092] This application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the method steps described in the above-described data job program creation method embodiment.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0094] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0095] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for creating a data job program, characterized in that, The method includes: Start and run the general framework program; When the framework program runs, it obtains configuration information for the target job program input by the user. The configuration information includes: basic information of the job program, field generation rules, and data filtering rules. The target job program is initialized according to the configuration information, the data corresponding to the target job program is read and stored, and the target job program is submitted and executed.

2. The data job program creation method according to claim 1, characterized in that, The basic information of the job procedure is stored in the job basic information table, which includes: job procedure name, job procedure type, data source information, and result table information; The field generation rules are stored in the field generation rule table, and the field generation rules include: the fields involved in the target job program and the methods for obtaining the fields; The data filtering rules are stored in the data filtering rule table.

3. The data job program creation method according to claim 2, characterized in that, The framework program includes at least: a job startup class, a field generation method class, and a data filtering class; The steps of initializing the target job program according to the configuration information, reading and storing the data corresponding to the target job program, and submitting and executing the target job program include: Load and run an object instance of the job startup class. When the object instance of the job startup class runs, it initializes the target job program and generates an input data transmission object based on the job program name, job program type, data source information and result table information in the job program basic information table. The input data transmission object is used to store input data. Load and run an object instance of the field generation method class. When the object instance of the field generation method class runs, it generates an output data transmission object based on the field generation rules in the field generation rule table and the input data transmission object, and the output data transmission object is used to store the original output data. Load and run an object instance of the data filtering class. When the object instance of the data filtering class runs, it filters the data in the output data object according to the data filtering rules in the data filtering rule table, and outputs the target data of the target job program according to the result table information. Submit and execute the initialized target job program and the target data.

4. The data job program creation method according to claim 3, characterized in that, The step of initializing the target job program and generating an input data transmission object based on the job program name, job program type, data source information, and result table information in the job program basic information table includes: The target job program is initialized based on the job program name, job program type, data source information, and result table information in the job program basic information table. Based on the data source information, determine the input data type; Based on the input data type, an input data transmission object is generated, and the input data indicated by the data source information is stored in the input data transmission object.

5. The data job program creation method according to claim 3, characterized in that, The step of generating an output data transmission object based on the field generation rules in the field generation rule table and the input data transmission object includes: According to the acquisition method in the field generation rule, obtain the field values ​​of each field in the field generation rule from the input data transmission object or related source; The output data transmission object is generated based on the field values ​​of each field.

6. The data job program creation method according to claim 5, characterized in that, The step of obtaining the field values ​​of each field in the field generation rule from the input data transmission object or a related source according to the acquisition method in the field generation rule includes: If the acquisition method is to obtain from the data source, then the field values ​​of each field in the field generation rule are obtained from the input data transmission object; If the acquisition method is through API, then the API identifier is obtained from the field generation rule table, the target API is initialized according to the API identifier, and the target API is called with each field in the field generation rule as input parameters to obtain the return result of the target API. The value in the return result is used as the field value of each field in the field generation rule.

7. The data job program creation method according to claim 3, characterized in that, The step of filtering the data in the output data object according to the data filtering rules in the data filtering rule table, and outputting the target data of the target job program according to the result table information, includes: The data in the output data object is filtered according to the data filtering rules in the data filtering rule table to obtain filtered data; Determine the data output type based on the information in the result table; The data output class corresponding to the data output type is called to output the filtered data according to the data output type, thereby obtaining the target data.

8. A data job program creation device, characterized in that, include: The startup module is used to start and run general framework programs; The acquisition module is used to acquire configuration information for the target job program input by the user during the runtime of the framework program. The configuration information includes: basic information of the job program, field generation rules, and data filtering rules. The submission module is used to initialize the target job program according to the configuration information, read and store the data corresponding to the target job program, and submit and execute the target job program.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program executable by the processor, and the processor executing the computer program to implement the steps of the data job program creation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data job program creation method as described in any one of claims 1-7.

Images