Data processing instruction submission method, data processing method, device and medium

Abstract

The present disclosure provides a data processing instruction submission method, a data processing method, an apparatus and a medium, relates to the technical field of computers, in particular to the field of big data and the field of data processing. The implementation scheme is: in response to receiving a submission request of a data processing instruction, a parameter configuration file is parsed to determine at least one environment configuration parameter; based on the at least one environment configuration parameter, a job running environment is configured; and in response to the running environment of the data processing client being configured to be completed, an instruction submission interface of the data processing client is called to submit the data processing instruction to the data processing client and make the data processing client execute the data processing instruction.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, in particular to the fields of big data and data processing, and more particularly to a data processing instruction submission method, a data processing method, a device, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] In the context of a surge in data volume, the value of data usually decreases over time after its generation. Stream data is a sequence of data that is sequential, massive, fast and continuously arriving. Generally, stream data can be regarded as a dynamic data set that grows indefinitely over time. Stream data is usually continuously generated by a large number of data sources.

[0003] With the development of Internet technology, Flink, as a framework and distributed processing engine, provides powerful processing capabilities. It can perform stateful computation on unbounded and bounded data streams, and Flink can run in all common cluster environments, perform computation at memory speed and any scale, and support the development and running of multiple different types of programs.

[0004] The methods described in this section can not necessarily be the methods previously conceived or adopted. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because it is included in this section. Similarly, issues mentioned in this section should not be assumed to have been recognized in any prior art unless otherwise indicated. SUMMARY

[0005] The present disclosure provides a data processing instruction submission method, a data processing method, a device, an electronic device, a computer readable storage medium and a computer program product.

[0006] According to an aspect of the present disclosure, a data processing instruction submission method is provided, comprising: in response to receiving a submission request of a data processing instruction, parsing a parameter configuration file to determine at least one environment configuration parameter, wherein the at least one environment configuration parameter is used to configure a job running environment, the parameter configuration file corresponds to the data processing instruction and includes at least one custom parameter used to determine the corresponding environment configuration parameter; based on the at least one environment configuration parameter, configuring the job running environment, wherein the job running environment includes a running environment of a data processing client, and the data processing client is used to execute the data processing instruction; and in response to the running environment of the data processing client being configured, calling an instruction submission interface of the data processing client to submit the data processing instruction to the data processing client and make the data processing client execute the data processing instruction.

[0007] According to another aspect of the present disclosure, a data processing method is provided, including: determining a data source end, a data target end and a data format corresponding to a data processing process; determining a data processing instruction and a parameter configuration file corresponding to the data processing process based on the data source end, the data target end and the data format, wherein the parameter configuration file corresponds to the data processing instruction and includes at least one custom parameter used for configuring a job running environment; in response to a submission request of the data processing instruction issued by a user, submitting the data processing instruction to a data processing client by an instruction submission client based on the parameter configuration file, wherein the instruction submission client is used for executing a submission method of the data processing instruction; and processing data from the data source end by the data processing client based on the data processing instruction to store processed data to the data target end.

[0008] According to another aspect of the present disclosure, a data processing instruction submission apparatus is provided, including: an analysis unit configured to analyze a parameter configuration file to determine at least one environment configuration parameter in response to receiving a submission request of a data processing instruction, wherein the at least one environment configuration parameter is used for configuring a job running environment, the parameter configuration file corresponds to the data processing instruction and includes at least one custom parameter used for determining the corresponding environment configuration parameter; a configuration unit configured to configure the job running environment based on the at least one environment configuration parameter, wherein the job running environment includes a running environment of a data processing client used for executing the data processing instruction; and a first submission unit configured to invoke an instruction submission interface of the data processing client to submit the data processing instruction to the data processing client and make the data processing client execute the data processing instruction in response to the running environment of the data processing client being configured to be completed.

[0009] According to another aspect of the present disclosure, a data processing apparatus is provided, including: a first determination unit configured to determine a data source end, a data target end and a data format corresponding to a data processing process; a second determination unit configured to determine a data processing instruction and a parameter configuration file corresponding to the data processing process based on the data source end, the data target end and the data format, wherein the parameter configuration file corresponds to the data processing instruction and includes at least one custom parameter used for configuring a job running environment; a second submission unit configured to submit the data processing instruction to a data processing client by an instruction submission client based on the parameter configuration file in response to a submission request of the data processing instruction issued by a user, wherein the instruction submission client is used for executing a submission method of the data processing instruction; and a processing unit configured to process data from the data source end by the data processing client based on the data processing instruction to store processed data to the data target end.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing instruction submission method or the data processing method.

[0011] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the data processing instruction submission method or the data processing method.

[0012] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the data processing instruction submission method or the data processing method.

[0013] According to one or more embodiments of the present disclosure, a user can be supported to configure a job running environment through a parameter configuration file, and the user can be supported to change a configuration parameter at any time during data processing, thereby improving flexibility of parameter configuration.

[0014] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are exemplary only and not limiting of the scope of the appended claims. In all the drawings, like reference numerals refer to like parts throughout the several views.

[0016] Figure 1 shows a schematic diagram of an exemplary system in which the various methods described herein can be implemented according to embodiments of the present disclosure;

[0017] Figure 2 shows a flowchart of a data processing instruction submission method according to embodiments of the present disclosure;

[0018] Figure 3 shows a flowchart of a data processing method according to embodiments of the present disclosure;

[0019] Figure 4 shows a flowchart of data processing according to exemplary embodiments of the present disclosure;

[0020] Figure 5A structural block diagram of a data processing instruction submission apparatus according to an embodiment of the present disclosure is shown;

[0021] Figure 6 A structural block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown;

[0022] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding them. These should be considered as merely exemplary. Thus, those skilled in the art will recognize that variations and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, in the following description, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0024] In the present disclosure, the terms "first", "second", and the like are used to describe various elements only and do not intend to limit the positional relationship, the chronological relationship, or the importance of the elements, and such terms are only used to distinguish one element from another. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0025] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the number of elements is specifically limited, the element can be one or more than one, unless the context clearly indicates otherwise. In addition, the term "and / or" used in the present disclosure encompasses any one of the listed items and all possible combinations thereof.

[0026] In the related art, in the field of big data, the submission of SQL (Structured Query Language) instructions for data processing is mainly performed by Shell script on a data processing client such as FlinkSQL or SparkSQL, and parameter settings are performed by Shell script. For complex SQL instructions, multiple processing and parameter settings are required in the middle, and Shell script can only set the global read-write end parameters, and cannot change the parameters during data processing.

[0027] According to an embodiment of the present disclosure, a data processing instruction submission method is provided. A parameter configuration file is read and parsed by a custom program, and a job running environment is configured based on the parsed configuration parameters. The parsed SQL instruction is submitted to the client through the instruction submission interface of the data processing client (FlinkSQL client) for execution. Thus, the user can configure the job environment through the parameter configuration file, and the user can change the configuration parameters at any time during the data processing process, thereby improving the flexibility of parameter adjustment.

[0028] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 A schematic diagram of an example system 100 in which various methods and apparatus described herein can be implemented according to embodiments of the present disclosure is shown. Referring to Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0030] In embodiments of the present disclosure, the server 120 can run one or more services or software applications that enable the execution of the above-described data processing instruction submission method or the above-described data processing method.

[0031] In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0032] In Figure 1 In the configuration shown, the server 120 can include one or more components that implement the functionality performed by the server 120. These components can include software components, hardware components, or a combination thereof, executable by one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client application programs to interact with the server 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible, which can differ from system 100. Thus, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0033] A user can use the client devices 101, 102, 103, 104, 105, and / or 106 to upload parameter configuration files and data processing instruction files. The client devices can provide an interface that enables a user of the client device to interact with the client device. The client devices can also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but those skilled in the art will appreciate that the present disclosure can support any number of client devices.

[0034] The client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0035] The networks 110 can be any type of networks known to those skilled in the art that can support data communications using any of a plurality of available protocols, including without limitation TCP / IP, SNA, IPX, etc. As examples only, one or more of the networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0036] The server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other appropriate arrangement and / or combination. The server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the servers). In various embodiments, the server 120 can run one or more services or software applications that provide the functionality described below.

[0037] The computing units in the server 120 can run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. The server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0038] In some embodiments, the server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of the client devices 101, 102, 103, 104, 105, and / or 106. The server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client devices 101, 102, 103, 104, 105, and / or 106.

[0039] In some embodiments, the server 120 can be a server of a distributed system, or a server combined with a blockchain. The server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0040] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as audio files and video files. The databases 130 can reside in various locations. For example, databases used by the server 120 can be local to the server 120 or can be remote from the server 120 and can communicate with the server 120 via a network- based or dedicated connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0041] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be different types of databases, such as key-value stores, object stores, or regular stores supported by file systems.

[0042] Figure 1 The system 100 can be configured and operated in various ways to enable the application of various methods and apparatuses described in accordance with the present disclosure.

[0043] According to some embodiments, as shown in Figure 2 a method of submitting a data processing instruction is provided, including: in response to receiving a submission request of a data processing instruction, parsing a parameter configuration file to determine at least one environment configuration parameter, where the at least one environment configuration parameter is used to configure a job running environment, the parameter configuration file corresponds to the data processing instruction and includes at least one custom parameter used to determine a corresponding environment configuration parameter; based on the at least one environment configuration parameter, configuring the job running environment, where the job running environment includes a running environment of a data processing client used to execute the data processing instruction; in response to the running environment of the data processing client being configured, calling an instruction submission interface of the data processing client to submit the data processing instruction to the data processing client and make the data processing client execute the data processing instruction.

[0044] Thus, the user can be supported to configure the job environment through the parameter configuration file, and the user can be supported to change the configuration parameter at any time during the data processing, improving the flexibility of parameter adjustment.

[0045] In some embodiments, the parameter configuration file includes at least one custom parameter set by the user for configuring the data processing job environment and the client running environment.

[0046] In some embodiments, the data processing instruction can be a SQL instruction.

[0047] In some embodiments, the data processing instruction can be obtained by parsing a data processing instruction file, and the data processing instruction file can be a SQL file including a SQL instruction for data processing.

[0048] In some embodiments, before preparing the SQL file and the parameter configuration file, it is necessary to first determine the data source end, the data target end and the data format of the data processing to be performed.

[0049] Different data source ends or data target ends differ in their data storage architectures. The data storage architecture may, for example, include a real-time Kafka or an offline Hive, HBase, Doris, MySQL, Oracle, etc. data storage architecture. The user can determine the storage framework of the data source end and the data target end respectively based on the data processing needs.

[0050] The data format includes structured, semi-structured and unstructured. For different data storage architectures and data formats, the corresponding SQL instruction for data processing is also different, so before writing the SQL file, the above information can be determined first.

[0051] In an example, the data storage architecture of the data source end can be determined as Kafka, the data storage architecture of the data target end can be determined as Hive, and the semi-structured data in the form of logs can be processed.

[0052] In some embodiments, based on the determined data storage architecture of the data source end, the data storage architecture of the data target end and the data format, the user can write the SQL file and determine the configuration parameters that need to be set.

[0053] According to some embodiments, the parameter configuration file can customize two types of parameters, i.e. running environment parameters and job environment parameters. The running environment parameters can be used to configure the running environment of the data processing client, and the job environment parameters can be used to configure the overall job environment of the data processing.

[0054] In some embodiments, the data processing client can be a FlinkSQL client, and specifically, for example, the FlinkSQL 1.11.0 version can be applied.

[0055] According to some embodiments, the running environment parameters can include at least one of the following: state backend parameters, checkpoint parameters and metadata management parameters.

[0056] The state backend parameter can be used to configure the state backend storage mode of the data processing client, determining the storage format and method of data during processing. In some examples, for the FlinkSQL client, the state backend parameter can be denoted as `state.backend`: that is, the Flink state backend, used to configure the location where persistent state data is stored, mainly in the following three ways:

[0057] 1) MemoryStateBackend: The memory-level state backend manages keyed states as objects in memory and stores them on the JVM (Java Virtual Machine) heap of the TaskManager. When a checkpoint is executed, a snapshot of the state is saved to the memory of the JobManager.

[0058] 2) FsStateBackend: Stores data based on a file system, such as a local file system or HDFS (Hadoop Distributed File System). When the state backend is set to FsStateBackend, the ongoing state data is still stored in the TaskManager's memory. Only during Checkpoint execution will a snapshot of the state data be written to the specified file system (such as HDFS).

[0059] 3) RocksDBStateBackend: Maintains the state in the local file system. The state is written directly to the local RocksDB. RocksDB needs to be configured with a remote filesystem.uri (usually HDFS). When performing a checkpoint, the local data is copied directly to the filesystem, and the data is restored from the filesystem to the local machine during failover.

[0060] The three backend settings mentioned above are applied to different scenarios. Memory-based storage (MemoryStateBackend) has the fastest processing speed, but the data cannot be persisted. FsStateBackend exists because the data needs to be persisted, but the corresponding processing performance is relatively low. RocksDB can support both persistent storage and efficient processing, but since it is not a component in big data, it requires multiple technologies for subsequent maintenance.

[0061] The checkpoint parameter can be used to configure the checkpoint of the data processing client, so as to periodically persist the state data based on the checkpoint, so that when the client crashes and restarts, the state data stored by the checkpoint can be used for recovery to correct the program data anomalies caused by the failure. In some examples, for the FlinkSQL client, the checkpoint parameter can include, for example, checkpoint.dir, checkpoint.interval (interval time of Checkpoint), checkpoint.timeout (timeout time of Checkpoint), and the like. When the value of state.backend is filesystem, checkpoint.dir needs to be configured, that is, the storage path of the state.

[0062] In some embodiments, the running environment parameter can also include table.exec.state.ttl, based on the setting of which, the keyed state defined in the job can be allowed to be automatically processed by timeout.

[0063] The metadata management parameter can be used to configure and establish the mapping relationship of the metadata between the data processing client and the data target end. The metadata management parameter can include, for example, a directory name, a configuration path, and the like. In one example, when the data processing client is a FlinkSQL client and the data target end is a Hive, the two parameters of hive.catalog.name (name of hive.catalog) and hive.config.path (configuration path of Hive) can be respectively customized. The hive.catalog.name is used for the registration of hive.catalog, and through the hive.catalog, the HiveMetaStore can be applied to manage the metadata of Flink, so as to establish the metadata mapping relationship between Flink and Hive. The hive.config.path can include MetaStore information, JDK information, storage engine information, Hadoop information, and the like, and is used to set the configuration path of Hive.

[0064] In addition, the parallelism, memory and other configuration information of the FlinkSQL client can also be customized.

[0065] Therefore, through the configuration of the above running environment parameters, the running environment of the FlinkSQL client can be flexibly configured.

[0066] According to some embodiments, the job environment parameter can include at least one of the following: source end memory, source end parallelism, target end memory, and target end parallelism.

[0067] In the related art, since the SQL instruction can only be submitted on the data processing client through the Shell script, the memory, parallelism and other parameters of the source end, the target end and the client can only be globally set, and the differentiated setting of each end cannot be realized, which makes the allocation of computing resources unable to be reasonably optimized, causing waste of computing resources and greatly affecting the efficiency of data processing.

[0068] Through the custom setting of the above job environment parameters in the parameter configuration file, and by calling the environment parameter configuration function of the corresponding function when configuring the environment, the differentiated setting of each end configuration is realized, the allocation of computing resources of each end is optimized, and the efficiency of data processing is improved.

[0069] In some embodiments, the parameter configuration file can also include main method parameters related to the data processing main method. Taking a data processing process with the source end as Kafka and the target end as Hive as an example, the main method parameters of the data processing can include, for example, the IP address, topic, username, password and other parameters of Kafka, and the IP address, topic, password and other parameters of Hive, which will also be used to configure the job running environment of data processing.

[0070] According to some embodiments, parsing the parameter configuration file to determine at least one environment configuration parameter can include: performing parameter verification on at least one custom parameter, wherein each custom parameter in the at least one custom parameter corresponds to a corresponding job running attribute of the job running environment; and in response to a first parameter in the at least one custom parameter passing the parameter verification, determining the first parameter as an environment configuration parameter of the corresponding job running attribute.

[0071] Since the configuration parameters customized by the user often have problems such as non-standard format and incorrect data type, it is necessary to perform parameter verification on the parameters, thereby ensuring the availability and reliability of the custom parameters.

[0072] In some embodiments, before reading each custom parameter in the parameter configuration file, it can be verified first whether the user has uploaded the parameter configuration file. In response to the user not uploading the corresponding file, the program can be directly ended. In response to verifying that the user has uploaded the parameter configuration file, the parameter configuration file can be read, and each custom parameter can be first subjected to parameter verification. As an example, the parameter verification can include, but is not limited to, verifying whether the data type of the parameter is consistent with the corresponding data type requirement, and verifying whether the data format of the parameter is consistent with the corresponding data format requirement.

[0073] In some embodiments, if it is found through verification that the data format of the verification parameter does not conform to the corresponding data format requirement, data conversion can be further performed. For example, if a certain configuration parameter is a date, the data format input by the user is "XXXX / XX / XX", and the required data format is "XXXXXXXX", data conversion can be performed on the data that does not conform to the requirement. In this way, the usability of the custom parameter can be further improved.

[0074] According to some embodiments, parsing the parameter configuration file to determine the at least one environment configuration parameter can further include: obtaining a default parameter file, wherein the default parameter file includes at least one default parameter, and each of the at least one default parameter corresponds to at least one job running attribute of the job running environment; and in response to a first attribute being included in the at least one job running attribute, determining a default parameter corresponding to the first attribute as the environment configuration parameter of the first attribute, wherein a custom parameter corresponding to the first attribute has not passed the parameter verification or is not given in the parameter configuration file.

[0075] In this way, for configuration parameters that have not passed the parameter verification or have not been custom set by the user, the corresponding default parameter value can be obtained, thereby avoiding the situation that the configuration parameter is missing and the environment configuration cannot be performed.

[0076] In some embodiments, the main method parameters related to the data processing main method can also be defined in the SQL file. In this case, when performing parameter parsing, the main method parameters in the SQL file also need to be verified. Specifically, before reading each main method parameter in the SQL, it is first verified whether the user has uploaded the SQL file. In response to the user not uploading the corresponding file, the program can be directly ended. In response to verifying that the user has uploaded the SQL file, the main method parameter value in the SQL file can be read by parsing the SQL file, and the parameter can be verified and converted based on a method similar to the above.

[0077] In some embodiments, the data processing instruction can be obtained by parsing a data processing instruction file, and the data processing instruction file can be a SQL file including SQL code for performing data processing.

[0078] In some embodiments, parsing the SQL file can first verify whether the user has uploaded the SQL file. In response to verifying that the user has uploaded the SQL file, the SQL file can be parsed. The specific operation can include, for example, identifying and deleting blank lines and comment lines, and merging the SQL code after deleting the blank lines and comment lines, thereby obtaining the parsed SQL file (i.e., the data processing instruction to be submitted).

[0079] According to some embodiments, the at least one environment configuration parameter can include a running environment parameter and a job environment parameter, and configuring the job running environment based on the at least one environment configuration parameter can include: configuring a running environment of the data processing client based on the running environment parameter; and configuring a job environment based on the job environment parameter, wherein the job environment includes a data source end environment and a data target end environment in a data processing process corresponding to the data processing instruction.

[0080] In this way, based on different parameters in the parameter configuration file, the data processing client running environment and the overall job environment (including the data source end and the data target end) can be configured respectively, thereby further improving the flexibility of environment configuration and solving the problem in the related art that the environment can only be configured as a whole based on global parameters.

[0081] In some embodiments, the running environment parameter can include at least one of the state backend parameter, the checkpoint parameter, and the metadata management parameter described above. Based on the above running environment parameters, the state backend, checkpoint, metadata management, and other related running environments of the FlinkSQL client can be configured.

[0082] In some examples, for a data processing process with a source end of Kafka and a target end of Hive, the running environment configuration can further include registering a Hive Category, thereby establishing a metadata mapping relationship between Flink and Hive through registration of the Hive Category.

[0083] In some embodiments, the job environment parameter can include at least one of the source end memory, the source end parallelism, the target end memory, and the target end parallelism described above. Based on the above job environment parameters, the available memory and the parallelism of the data source end, the data target end, and the data processing end can be set respectively, thereby optimizing the resource configuration of each end in the running environment and improving the data processing efficiency.

[0084] In some embodiments, in response to completing the configuration of the job environment and the running environment

[0085] In some embodiments, the above method can be programmed and encapsulated to encapsulate a software package for submitting SQL instructions, and the software package can be installed on a terminal operated by a user, thereby assisting the user in submitting SQL instructions.

[0086] Thus, by executing the submission of the SQL instruction through the software package, the user only needs to write the SQL file according to the requirement and upload the SQL file to the specified storage path, without other additional operations, so that the software package can automatically process the SQL instruction and submit the SQL instruction to the FlinkSQL client for execution, thereby simplifying the user operation and improving the user experience.

[0087] In some embodiments, through the above instruction submission method, the user can also adjust the environment configuration parameters at any time according to the actual needs during the data processing process. Specifically, the user can re-upload the parameter configuration file after updating the custom configuration parameters, and then re-run the instruction submission program, so as to reconfigure the running environment and the job environment without affecting the data processing process.

[0088] In some embodiments, the software package can also further encapsulate some custom UDF functions for data processing, such as functions for MD5 or RSA encryption of data, and register each UDF function in the running environment when configuring the running environment, thereby supporting the call of the custom UDF function by the SQL instruction. Thus, by encapsulating the custom UDF function in the software package in advance, the user does not need to master other non-SQL business codes for defining the UDF function, thereby further simplifying the user operation, saving the labor cost, and improving the user experience.

[0089] According to some embodiments, as shown in Figure 3 A data processing method is provided, including: step S301, determining a data source end, a data target end, and a data format corresponding to a data processing process; step S302, determining a data processing instruction file and a parameter configuration file corresponding to the data processing process based on the data source end, the data target end, and the data format, wherein the parameter configuration file corresponds to the data processing instruction and includes at least one custom parameter for configuring a job running environment; step S303, in response to a submission request of a data processing instruction issued by a user, submitting the corresponding data processing instruction to a data processing client through an instruction submission client based on the parameter configuration file and the data processing instruction file, wherein the instruction submission client is used to execute the above submission method of the data processing instruction; and step S304, processing data originating from the data source end through the data processing client based on the data processing instruction, to store the processed data to the data target end.

[0090] In some embodiments, the parameter configuration file includes at least one custom parameter set by the user for configuring the data processing job environment and the client running environment.

[0091] In some embodiments, the data processing instruction can be obtained by parsing a data processing instruction file, and the data processing instruction file can be an SQL file including SQL code for data processing.

[0092] In some embodiments, before preparing the SQL file and the parameter configuration file, the data source end, the data target end, and the data format of the data processing to be performed need to be determined first.

[0093] Different data source ends or data target ends differ in their data storage architectures. The data storage architecture can include, for example, a real-time Kafka or an offline Hive, HBase, Doris, MySQL, Oracle, etc. The user can determine the storage framework of the data source end and the data target end based on the data processing needs.

[0094] The data format includes structured, semi-structured, and unstructured. For different data storage architectures and data formats, the corresponding SQL instructions for data processing are also different. Therefore, before writing the SQL file, the above information should be determined first.

[0095] In an example, the data storage architecture of the data source end can be determined as Kafka, the data storage architecture of the data target end can be determined as Hive, and the semi-structured data in the form of logs can be processed.

[0096] After the SQL instruction is submitted to the configured FlinkSQL client through the above method, the client can call the Calcite tool (an open source SQL instruction parsing tool) to parse the submitted SQL instruction, and finally submit it to the Flink cluster for running to perform data processing.

[0097] Through the above data processing method, the user only needs to edit the SQL file and set the parameter configurations in the parameter configuration file. The SQL file can be parsed by the instruction submission client, and the SQL execution environment can be configured according to the parameter configuration file. Then the parsed SQL instruction can be uploaded to the FlinkSQL client for execution by one key. Thus, the user does not need to submit instructions through shell scripts on the FlinkSQL client, which is more convenient for the user to use. Further, the submission of the SQL instruction no longer depends on the FlinkSQL client, so the user does not need to consider the syntax difference of the shell script brought by different operating systems when submitting the SQL instruction, which further improves the user experience.

[0098] Figure 4 A flowchart of data processing according to an example embodiment of the present disclosure is shown.

[0099] As Figure 4 shown, the data processing flow can include: step S401, based on the instruction submission program, parsing the configuration parameters in the parameter configuration file uploaded by the user; step S402, verifying the configuration parameters obtained by parsing; step S403, performing parameter conversion on the configuration parameters to obtain environment configuration parameters; step S404, setting Checkpoint based on the environment configuration parameters; step S405, based on the environment configuration parameters and the custom UDF function encapsulated in the instruction submission program, registering the UDF function into the running environment; step S406, registering Hive Category based on the environment configuration parameters; step S407, parsing the SQL file to obtain the SQL instruction; step S408, calling the instruction submission interface of the FlinkSQL client to submit the SQL instruction to the FlinkSQL client; step S409, the FlinkSQL client parses the submitted SQL instruction by calling the Calcite tool and finally submits it to the Flink cluster; step S410, the Flink cluster performs distributed processing on the data from the data source end based on the parsed SQL instruction; and step S411, the Flink cluster stores the processed data to the data target end.

[0100] According to some embodiments, as Figure 5 shown, a data processing instruction submission apparatus 500 is provided, comprising: an analysis unit 510 configured to analyze a parameter configuration file to determine at least one environment configuration parameter in response to receiving a data processing instruction submission request, wherein the at least one environment configuration parameter is used to configure a job running environment, the parameter configuration file corresponds to the data processing instruction and includes at least one custom parameter used to determine the corresponding environment configuration parameter; a configuration unit 520 configured to configure the job running environment based on the at least one environment configuration parameter, wherein the job running environment includes a running environment of a data processing client used to execute the data processing instruction; and a first submission unit 530 configured to, in response to the running environment of the data processing client being configured to be completed, call an instruction submission interface of the data processing client to submit the data processing instruction to the data processing client and make the data processing client execute the data processing instruction.

[0101] Among them, the units 510-530 in the data processing instruction submission apparatus 500 perform operations similar to steps S201-S203 in the above data processing instruction submission method, and will not be repeated here.

[0102] Therefore, the user can configure the job environment through the parameter configuration file, and the user can change the configuration parameters at any time during the data processing, thereby improving the flexibility of parameter adjustment.

[0103] According to some embodiments, the parsing unit can include: a verification subunit configured to perform parameter verification on the at least one custom parameter, wherein each custom parameter in the at least one custom parameter corresponds to a corresponding job running attribute of the job running environment; and a first determination subunit configured to determine a first parameter in the at least one custom parameter as an environment configuration parameter of the corresponding job running attribute in response to the first parameter passing the parameter verification.

[0104] Since the user-defined configuration parameters often have problems such as non-standard format and incorrect data type, it is necessary to perform parameter verification on the parameters, thereby ensuring the availability and reliability of the custom parameters.

[0105] According to some embodiments, the parsing unit can further include: an obtaining subunit configured to obtain a default parameter file, wherein the default parameter file includes at least one default parameter, and the at least one default parameter respectively corresponds to at least one job running attribute of the job running environment; and a second determination subunit configured to determine a default parameter corresponding to a first attribute as an environment configuration parameter of the first attribute in response to the first attribute, wherein the custom parameter corresponding to the first attribute does not pass the parameter verification, or the custom parameter corresponding to the first attribute is not given in the parameter configuration file.

[0106] Therefore, for configuration parameters that do not pass the parameter verification or for which the user does not perform custom settings, the corresponding default parameter value of the configuration parameter can be obtained, thereby avoiding the case that the configuration parameter is missing and the environment configuration cannot be performed.

[0107] According to some embodiments, the at least one environment configuration parameter can include a running environment parameter and a job environment parameter, and the configuration unit can include: a first configuration subunit configured to configure a running environment of the data processing client based on the running environment parameter; and a second configuration subunit configured to configure a job environment based on the job environment parameter, wherein the job environment includes a data source end and a data target end in a data processing process, and the data processing process corresponds to a data processing instruction.

[0108] Therefore, based on different parameters in the parameter configuration file, the running environment of the data processing client and the overall job environment (including the data source end and the data target end) can be configured respectively, thereby further improving the flexibility of environment configuration and solving the problem in the related art that the environment can only be configured as a whole based on global parameters.

[0109] According to some embodiments, the running environment parameters can include at least one of the following: state backend parameters, checkpoint parameters, and metadata management parameters.

[0110] Thus, by configuring the above running environment parameters, flexible configuration of the running environment of the data processing client can be achieved.

[0111] According to some embodiments, the job environment parameters can include at least one of the following: source end memory, source end parallelism, target end memory, and target end parallelism.

[0112] Thus, by customizing the above job environment parameters in the parameter configuration file, and by calling the environment parameter configuration function of the corresponding function when performing environment configuration, the differentiated setting of each end configuration is achieved, the allocation of computing resources of each end is optimized, and thus the efficiency of data processing is improved.

[0113] According to some embodiments, as shown in Figure 6 A data processing apparatus 600 is provided, including: a first determination unit 610 configured to determine a data source end, a data target end, and a data format corresponding to a data processing process; a second determination unit 620 configured to determine a data processing instruction and a parameter configuration file corresponding to the data processing process based on the data source end, the data target end, and the data format, wherein the parameter configuration file corresponds to the data processing instruction and includes at least one custom parameter for configuring a job running environment; a second submission unit 630 configured to, in response to a submission request of the data processing instruction issued by a user, submit the data processing instruction to a data processing client through an instruction submission client based on the parameter configuration file, wherein the instruction submission client is used to execute a submission method of the above data processing instruction; and a processing unit 640 configured to process data originating from the data source end through the data processing client based on the data processing instruction, so as to store the processed data to the data target end.

[0114] Among them, the units 610-640 in the data processing instruction submission apparatus 600 perform operations similar to steps S301-S304 in the above data processing instruction submission method, and will not be repeated here.

[0115] Therefore, the user only needs to edit the SQL file and set the parameter configurations in the parameter configuration file, and the SQL file can be parsed by the instruction submission client, and the SQL execution environment is configured according to the parameter configuration file, and then the parsed SQL instruction is uploaded to the FlinkSQL client for execution. Thus, the user does not need to submit instructions through shell scripts on the FlinkSQL client, which is more convenient for the user to use; further, the submission of the SQL instruction no longer depends on the FlinkSQL client, so the user does not need to consider the syntax difference of the shell script caused by different operating systems when submitting the SQL instruction, which further improves the user experience.

[0116] According to embodiments of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0117] Reference Figure 7 The structure block diagram of the electronic device 700 which can be a server or a client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent a variety of forms of digital electronic computing devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent a variety of forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0118] As shown in Figure 7 , the electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0119] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device capable of inputting information to the electronic device 700, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device, and can include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote control. The output unit 707 can be any type of device capable of presenting information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0120] The computing unit 701 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the above-described data processing instruction submission method or the above-described data processing method. For example, in some embodiments, the above-described data processing instruction submission method or the above-described data processing method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the above-described data processing instruction submission method or the above-described data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the above-described data processing instruction submission method or the above-described data processing method by any other appropriate means, such as by means of firmware.

[0121] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0122] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0123] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0125] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0126] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0127] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0128] While embodiments or examples of this disclosure have been described with reference to the figures, it will be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the application is not limited to these embodiments or examples. Various elements of the embodiments or examples can be omitted or substituted by equivalents thereof. Furthermore, the steps can be performed in a different order than described in the disclosure. Further, various elements of the embodiments or examples can be combined in various ways. It is important that as technology evolves, many of the elements described herein can be substituted by equivalents which serve the same function.

Claims

1. A data processing instruction submission method performed by an instruction submission client, comprising: in response to receiving a submission request of a data processing instruction, parsing a parameter configuration file corresponding to the data processing instruction and comprising at least one custom parameter used to determine a corresponding environment configuration parameter, to determine at least one environment configuration parameter used to configure a job running environment and comprising a running environment parameter and a job environment parameter, comprising: performing parameter verification on the at least one custom parameter, each custom parameter corresponding to a corresponding job running attribute of the job running environment; in response to a first parameter in the at least one custom parameter passing the parameter verification, determining the first parameter as the environment configuration parameter of the corresponding job running attribute; obtaining a default parameter file comprising at least one default parameter, the at least one default parameter corresponding to at least one job running attribute of the job running environment respectively; in response to a first attribute in the at least one job running attribute comprising a custom parameter failing the parameter verification or the corresponding custom parameter not being given in the parameter configuration file, determining a default parameter of the first attribute as the environment configuration parameter of the first attribute; differentially configuring each end of the job running environment based on the at least one environment configuration parameter, comprising: configuring a running environment of a data processing client used to execute the data processing instruction based on the running environment parameter; configuring a data source end environment and a data target end environment in a data processing process corresponding to the data processing instruction based on the job environment parameter; and in response to the running environment of the data processing client being configured to be complete, calling an instruction submission interface of the data processing client to submit the data processing instruction to the data processing client and make the data processing client execute the data processing instruction.

2. The method of claim 1, wherein, The running environment parameter comprises at least one of the following: a state backend parameter, a checkpoint parameter, and a metadata management parameter.

3. The method of claim 1, wherein, The job environment parameter comprises at least one of the following: a source end memory, a source end parallelism, a target end memory, and a target end parallelism. 4.A data processing method, comprising: determining a data source end, a data target end, and a data format corresponding to a data processing process; based on the data source end, the data target end, and the data format, determining a data processing instruction and a parameter configuration file corresponding to the data processing process, wherein the parameter configuration file corresponds to the data processing instruction and comprises at least one custom parameter used to configure a job running environment; in response to a user issuing a submission request of the data processing instruction, submitting the data processing instruction to a data processing client through an instruction submission client based on the parameter configuration file, wherein the instruction submission client is used to execute the method of any one of claims 1-3; and based on the data processing instruction, processing data originating from the data source end through the data processing client to store the processed data to the data target end.

5. An apparatus for submitting a data processing instruction based on a client, the apparatus comprising: a parsing unit configured to, in response to receiving a submission request of a data processing instruction, parse a parameter configuration file corresponding to the data processing instruction and comprising at least one customized parameter for determining a corresponding environment configuration parameter, to determine at least one environment configuration parameter for configuring a job running environment and comprising a running environment parameter and a job environment parameter, the parsing unit comprising: a verification sub-unit configured to perform parameter verification on the at least one customized parameter, each customized parameter corresponding to a corresponding job running attribute of the job running environment; a first determination sub-unit configured to, in response to a first parameter in the at least one customized parameter passing the parameter verification, determine the first parameter as an environment configuration parameter of the corresponding job running attribute; an acquisition sub-unit configured to acquire a default parameter file comprising at least one default parameter, the at least one default parameter corresponding to at least one job running attribute of the job running environment, respectively; and a second determination sub-unit configured to, in response to a first attribute in the at least one job running attribute comprising a customized parameter failing the parameter verification or the corresponding customized parameter not being given in the parameter configuration file, determine a default parameter of the first attribute as an environment configuration parameter of the first attribute; a configuration unit configured to, based on the at least one environment configuration parameter, perform differential configuration on each end of the job running environment, the configuration unit comprising: a first configuration sub-unit configured to, based on the running environment parameter, configure a running environment of a data processing client for executing the data processing instruction; and a second configuration sub-unit configured to, based on the job environment parameter, configure a data source end environment and a data target end environment in a data processing process corresponding to the data processing instruction; and a first submission unit configured to, in response to the running environment of the data processing client being configured to be completed, invoke an instruction submission interface of the data processing client to submit the data processing instruction to the data processing client and make the data processing client execute the data processing instruction.

6. The apparatus of claim 5, wherein, The running environment parameter comprises at least one of the following: a state backend parameter, a checkpoint parameter, and a metadata management parameter.

7. The apparatus of claim 5, wherein, The job environment parameter comprises at least one of the following: a source end memory, a source end parallelism, a target end memory, and a target end parallelism.

8. An apparatus for data processing, the apparatus comprising: a first determination unit configured to determine a data source end, a data target end, and a data format corresponding to a data processing process; a second determination unit configured to, based on the data source end, the data target end, and the data format, determine a data processing instruction and a parameter configuration file corresponding to the data processing process, wherein the parameter configuration file corresponds to the data processing instruction and comprises at least one customized parameter for configuring a job running environment; a second submitting unit, configured to, in response to a submitting request of the user issuing the data processing instruction, submit the data processing instruction to a data processing client based on the parameter configuration file through an instruction submitting client, wherein the instruction submitting client is used to execute the method in any one of claims 1-3; and a processing unit, configured to process data from the data source end through the data processing client based on the data processing instruction, and store the processed data to the data target end.

9. An electronic device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-4.

11. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-4. The computer program, when executed by a processor, implements the method of any one of claims 1-4.

Images