Data processing method, device, electronic device and computer readable medium

Abstract

The present invention discloses a data processing method, device, electronic device and computer-readable medium, and relates to the field of big data technology. The specific implementation of the method includes: obtaining a data processing request, the data processing request indicates the data to be processed and the processing method of the data to be processed; determining one or more common processing logics included in the processing method; obtaining a task template according to one or more common processing logics; generating a data processing task according to the task template and the data to be processed; executing the data processing task in response to the data processing request. This implementation can extract the common processing logic from the task logic, reduce the complexity of the task logic, improve the generation efficiency of the task, and thus improve the efficiency of big data processing.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, device, electronic device and computer-readable medium. Background Art

[0002] With the development of computer and network technology, all walks of life are gradually moving towards informatization, which has resulted in massive data resources. In order to make rational use of data resources, the analysis and processing of big data is particularly important.

[0003] In the existing big data processing process, different processing tasks need to be generated for different processing requirements. The logic of each processing task is different. When the task is executed, it is necessary to conduct a specific analysis of the logic of each task and execute the processing task.

[0004] The logic of existing processing tasks usually has similar parts. In the process of generating each task, similar processing logic needs to be added to the task script, resulting in low task generation efficiency and correspondingly low big data processing efficiency. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data processing method, device, electronic device and computer-readable medium, which can extract common processing logic from task logic, reduce the complexity of task logic, improve task generation efficiency, and thereby improve big data processing efficiency.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0007] Obtaining a data processing request, wherein the data processing request indicates the data to be processed and a processing method for the data to be processed;

[0008] Determine one or more common processing logics included in the processing method;

[0009] According to the one or more common processing logics, obtaining a task template;

[0010] Generate a data processing task according to the task template and the data to be processed;

[0011] The data processing task is executed in response to the data processing request.

[0012] Optionally, when the plurality of common processing logics correspond to a plurality of task templates,

[0013] Determine whether there is a dependency relationship between the multiple task templates;

[0014] If so, the data processing tasks corresponding to the multiple task templates are executed in sequence according to the dependency relationship.

[0015] Optionally, the task template is configured with task dependency information;

[0016] It is determined whether the plurality of task templates have the dependency relationship according to the task dependency information respectively included in the plurality of task templates.

[0017] Optionally, when the plurality of task templates do not have the dependency relationship,

[0018] The data processing tasks respectively corresponding to the multiple task templates are executed in parallel.

[0019] Optionally, generating a data processing task according to the task template and the data to be processed includes:

[0020] Determine, according to the data to be processed, feature processing logic other than the common processing logic;

[0021] The data processing task is generated according to the data to be processed corresponding to the feature processing logic, the data to be processed corresponding to the common processing logic, and the task template.

[0022] Optionally, generating the data processing task according to the data to be processed corresponding to the feature processing logic, the data to be processed corresponding to the common processing logic, and the task template includes:

[0023] Determine a first script corresponding to the task template and the feature processing logic according to the feature processing logic, the data to be processed corresponding to the feature processing logic, the common processing logic corresponding to the task template, and the data to be processed corresponding to the task template;

[0024] The data processing task is generated according to the first script.

[0025] Optionally, it also includes:

[0026] Get historical execution data of multiple historical tasks;

[0027] Determining, based on the historical execution data, a common processing logic shared by the plurality of historical tasks;

[0028] A second script corresponding to the common processing logic shared by the plurality of historical tasks is generated, and the task template is generated according to the second script.

[0029] Optionally, the task template also includes: task description information, task parameter information, and task scheduling information; wherein the task parameter information includes any one or more of the following: task type, storage path, and execution thread.

[0030] Optionally, the task parameter information includes: a data source identifier corresponding to the task template;

[0031] The data source corresponding to the data source identifier is monitored, and when it is detected that the name of the data source has changed, the data source identifier in the task template is updated.

[0032] Optionally, different task types correspond to different task templates.

[0033] Optionally, the data processing request further indicates a time parameter corresponding to the data to be processed; the task template includes: a time parameter format;

[0034] Converting the time parameter into a target format corresponding to the time parameter format;

[0035] According to the target format, the to-be-processed data corresponding to the time parameter is acquired from a database.

[0036] Optionally, before generating the data processing task, the method further includes:

[0037] Obtaining a configuration file, wherein the configuration file includes a system configuration file and an application configuration file;

[0038] According to the configuration file, the to-be-processed data corresponding to the time parameter is acquired.

[0039] According to another aspect of an embodiment of the present invention, there is provided a data processing device, including:

[0040] An acquisition module, used for acquiring a data processing request, wherein the data processing request indicates the data to be processed and a processing method of the data to be processed;

[0041] A logic processing module, used to determine one or more common processing logics included in the processing method;

[0042] The acquisition module is further used to acquire the task template according to the one or more common processing logics;

[0043] A task generation module, used to generate a data processing task according to the task template and the data to be processed;

[0044] An execution module is used to execute the data processing task in response to the data processing request.

[0045] According to another aspect of an embodiment of the present invention, there is provided a data processing electronic device, comprising:

[0046] one or more processors;

[0047] a storage device for storing one or more programs,

[0048] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided by the present invention.

[0049] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data processing method provided by the present invention is implemented.

[0050] An embodiment of the above invention has the following advantages or beneficial effects: because it adopts a technical means of generating data processing tasks by determining the common processing logic shared by multiple data processing tasks based on historical data, and generating a task template based on the script of the common processing logic, combining the script corresponding to the feature processing logic corresponding to the data processing request, it overcomes the technical problems of low task generation efficiency and low data processing efficiency, thereby achieving the technical effect of extracting the common processing logic from the task logic, reducing the complexity of the task logic, and improving the task generation efficiency, thereby improving the efficiency of big data processing.

[0051] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0053] Figure 1 An exemplary system architecture diagram of a data processing method or a data processing device suitable for application in an embodiment of the present invention is shown;

[0054] Figure 2 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of a format class operator according to an embodiment of the present invention;

[0056] Figure 4 is a schematic diagram of a calculation operator according to an embodiment of the present invention;

[0057] Figure 5 is a schematic diagram of the main flow of a method for acquiring to-be-processed data of time parameters in a target format according to an embodiment of the present invention;

[0058] Figure 6 is a schematic diagram of the main process of the task template generation method according to an embodiment of the present invention;

[0059] Figure 7 is a schematic diagram of the main process of the method for generating a data processing task according to an embodiment of the present invention;

[0060] Figure 8 is a schematic diagram of the main process of the task execution method according to an embodiment of the present invention;

[0061] Fig. 9 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0062] Fig.10 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0064] Figure 1 An exemplary system architecture diagram of a data processing method or a data processing device suitable for application in an embodiment of the present invention is shown. Figure 1 As shown, an exemplary system architecture of the data processing method or data processing device according to an embodiment of the present invention includes:

[0065] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0066] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as financial applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0067] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0068] The server 105 may be a server that provides various services, such as a background management server that provides support for financial application websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as the data processing task generation request, and feed back the processing results (such as the data processing task) to the terminal devices 101, 102, and 103.

[0069] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 105 , and accordingly, the data processing device is generally disposed in the server 105 .

[0070] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0071] Figure 2 is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention, such as Figure 2 As shown, the data processing method of the present invention includes:

[0072] Multiple data processing tasks usually include some of the same processing logic, namely the common processing logic, and each data processing task usually includes its own unique processing logic, namely the feature processing logic. The data processing method of the present invention extracts the common processing logic from the entire task logic and constructs a task template. When processing data in the later stage, the data processing task is generated and executed by filling in the data corresponding to the common processing logic and the data corresponding to the feature processing logic, thereby greatly improving the data processing efficiency.

[0073] Step S201: obtaining a data processing request, wherein the data processing request indicates the data to be processed and a processing method for the data to be processed.

[0074] In an embodiment of the present invention, the data processing request also indicates a time parameter corresponding to the data to be processed. The time parameter may be one or more of the data to be processed, or the time parameter may also be a file time attribute of a file corresponding to the data to be processed. For example, one or more files corresponding to the data to be processed were created / acquired on January 1, 2021, and the file time attribute of the file is 20200101.

[0075] Step S202: determine one or more common processing logics included in the processing method.

[0076] In the embodiment of the present invention, the common processing logic refers to the script corresponding to the same logic part in the processing logic indicated by the processing mode. The common processing logic can be determined by historical data.

[0077] Step S203: acquiring a task template according to the one or more common processing logics.

[0078] In the embodiment of the present invention, the task template is configured with task description information, task parameter information, task dependency information, and task scheduling information.

[0079] In an embodiment of the present invention, the task parameter information may include a data source identifier corresponding to the task template, monitor the data source corresponding to the data source identifier, and update the data source identifier in the task template when it is detected that the name of the data source has changed. For example, the data source identifier in the task parameter information in the task template is Table A, indicating that the data source of the data to be processed is Table A; Table A is monitored, and when it is detected that the name of Table A has changed to Table A', the data source identifier Table A in the task template is updated to Table A'.

[0080] In an embodiment of the present invention, the task parameter information of the task template may include the time parameter format and the conversion of the time parameter format. The time parameter format of the task parameter information may be ${now.XX.XX…XX.XX}. Among them, "XX" represents each operator; each operator is separated by a dot "."; the operator string starts with "now", which represents the currently determined time parameter, and the "XX.XX…XX.XX" after now represents the specific content of the operator string; each operator "XX" in the middle is a calculation operator, which can be omitted, and multiple calculation operators can be used in combination; the last operator "XX" is a format operator, which can be omitted, and the operators after the first format operator are invalid.

[0081] Furthermore, if Figure 3 As shown, the format class operator of the present invention includes:

[0082] week_begin is the timestamp of the same time on Monday of the current week, indicating the same time on Monday of the current week corresponding to the current time;

[0083] week_end is the timestamp of the same time on Sunday of the current week, indicating the same time on Sunday of the current week corresponding to the current time;

[0084] month_begin is the timestamp of the same time as the beginning of the month, indicating the same time as the current time. For example, if the current time is 2020-03-31-xx-xx-xx, month_begin is 2020-03-01-xx-xx-xx.

[0085] month_end is the timestamp of the end of the month, indicating the same time as the end of the month corresponding to the current time; for example, if the current time is 2020-03-13-xx-xx-xx, month_end is 2020-03-31-xx-xx-xx;

[0086] year_begin is the timestamp of the beginning of the year, indicating the same time as the beginning of the year corresponding to the current time; for example, if the current time is 2020-03-13-xx-xx-xx, year_begin is 2020-01-01-xx-xx-xx;

[0087] year_end is the timestamp of the year-end date, indicating the same time as the year-end date of the current time; for example, if the current time is 2020-03-13-xx-xx-xx, year_end is 2020-12-31-xx-xx-xx;

[0088] quarter_begin is the timestamp of the beginning of the quarter, indicating the same time as the beginning of the quarter corresponding to the current time; for example, if the current time is 2020-03-13-xx-xx-xx, quarter_begin is 2020-01-01-xx-xx-xx;

[0089] quarter_end is the timestamp of the quarter end date, indicating the same time as the quarter end date corresponding to the current time; for example, if the current time is 2020-03-13-xx-xx-xx, quarter_end is 2020-03-31-xx-xx-xx;

[0090] delta(days=1) is the timestamp of the previous day, abbreviated as delta(1), which means the same time of the previous day corresponding to the current time. For example, if the current time is 2020-03-13-xx-xx-xx, delta(1) is 2020-03-12-xx-xx-xx.

[0091] delta(months=1) is the timestamp of the same day and time of the previous month, indicating the same time of the same day of the previous month corresponding to the current time; for example, if the current time is 2020-03-13-xx-xx-xx, delta(months=1) is 2020-02-13-xx-xx-xx;

[0092] If this day does not exist in the previous month, it is the last day of the previous month; for example, the current time is 2020-03-31-xx-xx-xx, and the same day of the previous month should be 2020-02-31, but there is no 31st in the previous month, so delta(months=1) is 2020-02-29-xx-xx-xx;

[0093] delta(years=1) is the timestamp of the same month, day and time last year, indicating the same day and time of the same month last year corresponding to the current time; for example, if the current time is 2020-03-13-xx-xx-xx, delta(years=1) is 2019-03-13-xx-xx-xx;

[0094] If this day does not exist in the same month last year, it is the last day of the same month last year; for example, the current time is 2020-02-29-xx-xx-xx, and the same day of the same month last year should be 2019-02-29, but there is no 29th in February 2019, so delta(years=1) is 2019-02-28-xx-xx-xx;

[0095] current indicates that the previous calculation operator is invalid; for example, ${now.delta(10).current} is equivalent to ${now.current}.

[0096] Furthermore, if Figure 4 As shown, the calculation operators of the present invention include:

[0097] time is a time string; for example, the value of time is 2020-09-14 17:39:46;

[0098] date is in the format of yyyy-MM-dd; for example, the value of date is 2020-09-14;

[0099] datekey is a date in the yyyyMMdd format; for example, the value of datekey is 20200914;

[0100] month is a date in the yyyy-MM format; for example, the value of month is 2020-09;

[0101] monthkey is a date in the yyyyMM format; for example, the value of monthkey is 202009;

[0102] year is the year; for example, the value of year is 2020;

[0103] quarter_of_year is the quarter number, and the value range is 1-4; for example, the value of quarter_of_year is 3;

[0104] month_of_year is the month number, and the value range is 1-12; for example, the value of month_of_year is 9;

[0105] month_of_year_format is the formatted month, and the value range is 01-12; for example, the value of month_of_year_format is 09;

[0106] day_of_year is the day of the year, and the value range is 1-366; for example, the value of day_of_year is 258;

[0107] day_of_year_format is the formatted day of the year, and the value range is 001-366; for example, the value of day_of_year_format is 258;

[0108] month_of_quarter is the month number in a quarter, ranging from 1 to 3; for example, the value of month_of_quarter is 1;

[0109] day_of_quarter is the day of a quarter, ranging from 1 to 62; for example, the value of day_of_quarter is 14;

[0110] day_of_quarter_format is the day of the quarter in the format, ranging from 01 to 62; for example, the value of day_of_quarter_format is 14;

[0111] day_of_month is the day of the month, ranging from 1 to 31; for example, the value of day_of_month is 14;

[0112] day_of_month_format is the formatted day of the month, and the value range is 01-31; for example, the value of day_of_month_format is 14;

[0113] day_of_week is the first day of the week and ranges from 1 to 7; for example, the value of day_of_week is 1.

[0114] In the embodiment of the present invention, the meaning represented by the time parameter operator string may be:

[0115] ${now.date} represents the current date in the yyyy-MM-dd format, for example, 2021-03-31;

[0116] ${now.delta(1).datekey} represents the date in yyyyMMdd format one day before the current time, for example, 2021-03-30;

[0117] ${now.delta(months=1).month_end.datekey} represents the date in yyyyMMdd format of the end of the previous month corresponding to the current time, for example, 2021-02-28;

[0118] ${now.day_of_year} indicates the current day of the year. For example, 2021-03-31-xx-xx-xx is the 90th day of the year.

[0119] In ${now.delta(1).datekey.delta(months=1).date}, since datekey is a format operator, operators after the first format operator are invalid. Therefore, the operator "delta(months=1).date" is invalid.

[0120] In ${now.delta(10).current}, since current indicates that the previous calculation operator is invalid, the operator "delta(10)" is invalid.

[0121] In the embodiment of the present invention, Figure 5 As shown, the method for acquiring the to-be-processed data of the time parameter of the target format of the present invention includes:

[0122] Step S501, determining a target format of a time parameter format.

[0123] In an embodiment of the present invention, the data source of the data to be processed can be in any time parameter format, including: (1) the data in the data source includes data in any time parameter format; (2) the file time attribute of the file corresponding to the data source is in any time parameter format.

[0124] Step S502: convert the time parameter into a target format corresponding to the time parameter format.

[0125] In an embodiment of the present invention, according to the conversion rules of the time parameter format, the time parameters in any time parameter format in the data source are converted into the time parameters in the target format, and / or the time parameters corresponding to the file time attributes are converted into the time parameters in the target format.

[0126] Step S503: acquiring the to-be-processed data corresponding to the time parameter from the database according to the target format.

[0127] In an embodiment of the present invention, the data corresponding to the time parameters in the database can be in any format. After determining the target format of the time parameters in the task parameter information, the time parameters in the database are converted into time parameters in the target format, so that the data to be processed corresponding to the time parameters can be obtained.

[0128] In the embodiment of the present invention, by converting any time parameter format into a unified target format, the time parameters can be unified and the data processing efficiency can be improved.

[0129] In the embodiment of the present invention, the task parameter information includes any one or more of the following: task type, storage path, and execution thread. Different task types correspond to different task templates.

[0130] Furthermore, the task templates of some task types are also configured with task result information and task process information.

[0131] In the embodiment of the present invention, Figure 6 As shown, the task template generation method of the present invention includes:

[0132] Step S601, obtaining historical execution data of multiple historical tasks;

[0133] Step S602: Determine the common processing logic shared by multiple historical tasks based on the historical execution data.

[0134] Step S603, generating a script corresponding to a common processing logic shared by a plurality of historical tasks, and generating a task template according to the script corresponding to the common processing logic.

[0135] In an embodiment of the present invention, through the task template generation method of the present invention, the common public processing logic is generated into a task template, so that the public processing logic is extracted from the task logic. There is no need to develop the same public processing logic in the script of each data processing task. Only the unique feature processing logic of the data processing task needs to be developed, which greatly improves the generation efficiency of the data processing task.

[0136] In the embodiment of the present invention, the task description information is used to describe the scenario and information of the data processing task, and can be represented by the desc area in the task template. For example, the desc area of ​​the data processing task includes a desc identifier. The value of the desc identifier can be querying x database x table, calculating the total sales volume of x product in 2016, and calculating the total sales volume of x product in 2016.

[0137] In an embodiment of the present invention, task parameter information is used to represent parameter information of a data processing task through variables, including system variables and custom variables, etc., which can be represented by the profile area in the task template. For example, the profile area of ​​a data processing task includes system variables and custom variables, and the system variables may include storage paths, execution threads, file names of target files, task types, time parameter formats, target formats, time parameter format conversions, etc. Among them, the value of the task type can be the conversion of an Excel table to a GP file, an Excel table to a TXT file, a GP file to a hive table, data type conversion within a hive table, and a hive table to an HBASE table. Furthermore, the task type can be defined by task.type.

[0138] In an embodiment of the present invention, task dependency information is used to represent dependency information of a data processing task, and can be represented by the dependencies area in a task template. For example, the dependencies area of ​​a data processing task includes a dependencies identifier, and the value of the dependencies identifier can be desc1, or desc1 and desc5. The current data processing task may need to perform data processing based on the execution results of the previous or previous data processing tasks, so there is a dependency relationship between data processing tasks.

[0139] In the embodiment of the present invention, the task scheduling information is used to represent the scheduling information of the data processing task, which can be represented by the schedule area in the task template. For example, the schedule area of ​​the data processing task includes a scheduling type and a scheduling parameter, and the value of the scheduling type can be times / week or times / month, and a thread can be allocated to the data processing task according to the task scheduling information.

[0140] In the embodiment of the present invention, the task result information is used to represent the result information of the data processing task, and can be represented by the targetDDL area in the task template. For example, the targetDDL area of ​​the data processing task includes a target library, and the value of the target library can be kv database 1, indicating that kv database 1 is established and the execution result is saved to kv database 1. Furthermore, the targetDDL area can also include a target table of the target library, indicating that a target table is established in the target library and the execution result is saved to the target table.

[0141] In an embodiment of the present invention, task flow information is used to represent the flow information of a data processing task, and can be represented by a preload area, a load area, and an afterload area in a task template. For example, the preload area of ​​a data processing task represents a data preprocessing area, and can include a preload identifier, and the value of the preload identifier can be data cleaning, data integration, data transformation, and data reduction, etc.; wherein, data cleaning means that the data to be processed is cleaned before the data processing task is executed. For another example, the load area of ​​a data processing task represents a data loading area, and can include a load identifier, and the value of the load identifier can be a loading method, etc. For another example, the afterload area of ​​a data processing task represents a data post-processing area, and can include an afterload identifier, and the value of the afterload identifier can be clearing temporary tables, deleting invalid partitions, etc.; wherein, clearing temporary tables means clearing temporary tables constructed during the data processing process after the data processing task is executed. Task flow information can reference custom variables in task parameter information.

[0142] In an embodiment of the present invention, the task template may also be configured with task verification information and task log information. Task verification information is used to indicate verification information of the data processing task, for example, the task verification information may include data accuracy verification standards, data timeliness verification standards, etc.; task log information is used to indicate log information of the data processing task, for example, the task log information may include log records, log analysis, log queries, etc.

[0143] Step S204: Generate a data processing task according to the task template and the data to be processed.

[0144] In an embodiment of the present invention, before generating a data processing task according to a task template and the data to be processed, a configuration file is obtained, and the data to be processed corresponding to the time parameter is obtained according to the configuration file. The configuration file includes a system configuration file and an application configuration file; the system configuration includes a system directory, a log storage database, etc.

[0145] In the embodiment of the present invention, Figure 7As shown, the data processing task generation method of the present invention includes:

[0146] Step S701, determining feature processing logic other than common processing logic according to the data to be processed.

[0147] Step S702, determining a script corresponding to the task template and the feature processing logic according to the feature processing logic, the data to be processed corresponding to the feature processing logic, the common processing logic corresponding to the task template, and the data to be processed corresponding to the task template;

[0148] Step S703: Generate a data processing task according to the task template and the script corresponding to the feature processing logic.

[0149] In an embodiment of the present invention, through the data processing task generation method of the present invention, based on the extraction of common processing logic from task logic, only characteristic processing logic unique to the data processing task needs to be developed, and data processing tasks are generated in combination with task templates, thereby greatly improving the generation efficiency of data processing tasks; wherein, the task template corresponds to the common processing logic.

[0150] Step S205: execute the data processing task in response to the data processing request.

[0151] In the embodiment of the present invention, after the data processing task is generated, the data processing task is executed according to the execution thread indicated by the data processing task.

[0152] In the embodiment of the present invention, when multiple common processing logics correspond to multiple task templates, such as Figure 8 As shown, the task execution method of the present invention includes:

[0153] Step S801 , determining whether there is a dependency relationship between the multiple task templates according to the task dependency information respectively included in the multiple task templates, if yes, go to step 802 ; if no, go to step 803 .

[0154] Step 802: Execute the data processing tasks corresponding to the multiple task templates in sequence according to the dependency relationship.

[0155] Step 803: execute the data processing tasks corresponding to the multiple task templates in parallel.

[0156] In the embodiment of the present invention, through the task execution method of the present invention, data processing tasks with dependencies are executed in sequence, and data processing tasks without dependencies are executed in parallel, which greatly improves the execution efficiency of data processing tasks.

[0157] In an embodiment of the present invention, by obtaining a data processing request, the data processing request indicates the data to be processed and the processing method of the data to be processed; determining one or more common processing logics included in the processing method; obtaining a task template according to the one or more common processing logics; generating a data processing task according to the task template and the data to be processed; executing the data processing task to respond to the data processing request and other steps, the common processing logic can be extracted from the task logic, the complexity of the task logic can be reduced, the task generation efficiency can be improved, and thus the big data processing efficiency can be improved.

[0158] Fig. 9 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention, such as Fig. 9 As shown, the data processing device 900 of the present invention includes:

[0159] The acquisition module 901 is used to acquire a data processing request, wherein the data processing request indicates the data to be processed and the processing method of the data to be processed.

[0160] In the embodiment of the present invention, the acquisition module 901 is used to acquire a data processing request, which indicates the data to be processed, the processing method of the data to be processed, the time parameter corresponding to the data to be processed, etc.

[0161] The logic processing module 902 is used to determine one or more common processing logics included in the processing method.

[0162] In the embodiment of the present invention, the logic processing module 902 is used to determine the common processing logic included in the processing method, and the common processing logic refers to the script corresponding to the same logic part in the processing logic indicated by the processing method. The common processing logic can be determined by historical data.

[0163] The acquisition module 901 is further used to acquire a task template according to the one or more common processing logics.

[0164] In the embodiment of the present invention, the acquisition module 901 is further configured to acquire a task template according to one or more common processing logics.

[0165] In the embodiment of the present invention, a task template generating module 905 is further included, which is used to generate a task template, including:

[0166] Get historical execution data of multiple historical tasks;

[0167] Based on the historical execution data, the common processing logic shared by multiple historical tasks is determined.

[0168] Generate a script corresponding to the common processing logic shared by multiple historical tasks, and generate a task template based on the script corresponding to the common processing logic.

[0169] In the embodiment of the present invention, the task template is configured with task description information, task parameter information, task dependency information, and task scheduling information.

[0170] The task generation module 903 is used to generate a data processing task according to the task template and the data to be processed.

[0171] In the embodiment of the present invention, the task generation module 903 is used to generate a data processing task, including:

[0172] According to the data to be processed, feature processing logic other than the common processing logic is determined.

[0173] Determine the script corresponding to the task template and the feature processing logic according to the feature processing logic, the data to be processed corresponding to the feature processing logic, the common processing logic corresponding to the task template, and the data to be processed corresponding to the task template;

[0174] Generate data processing tasks based on the script corresponding to the task template and feature processing logic.

[0175] In the embodiment of the present invention, a configuration file acquisition module 906 is also included, which is used to acquire a configuration file before generating a data processing task according to the task template and the data to be processed, and acquire the data to be processed corresponding to the time parameter according to the configuration file. The configuration file includes a system configuration file and an application configuration file; the system configuration includes a system directory, a log storage database, etc.

[0176] The execution module 904 is used to execute the data processing task in response to the data processing request.

[0177] In the embodiment of the present invention, after the data processing task is generated, the execution module 904 executes the data processing task according to the execution thread indicated by the data processing task.

[0178] In an embodiment of the present invention, through modules such as an acquisition module, a logic processing module, a task generation module and an execution module, the common processing logic can be extracted from the task logic, the complexity of the task logic can be reduced, the task generation efficiency can be improved, and the big data processing efficiency can be improved.

[0179] Fig.10 is a schematic diagram of the structure of a computer system of a terminal device suitable for implementing an embodiment of the present invention, such as Fig.10 As shown, the computer system 1000 of the terminal device of the embodiment of the present invention includes:

[0180] A central processing unit (CPU) 1001 can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage section 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the system 1000 are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0181] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0182] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present invention are executed.

[0183] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0184] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0185] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: a processor including an acquisition module, a logic processing module, a task generation module and an execution module. The names of these modules do not constitute a limitation on the modules themselves in certain cases, for example, the task generation module may also be described as "a module for generating data processing tasks according to task templates and data to be processed".

[0186] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining a data processing request, the data processing request indicating the data to be processed and the processing method of the data to be processed; determining one or more common processing logics included in the processing method; obtaining a task template according to the one or more common processing logics; generating a data processing task according to the task template and the data to be processed; and executing the data processing task in response to the data processing request.

[0187] According to the technical solution of the embodiment of the present invention, the common processing logic can be extracted from the task logic, the complexity of the task logic can be reduced, the task generation efficiency can be improved, and the big data processing efficiency can be improved.

[0188] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Obtaining a data processing request, wherein the data processing request indicates data to be processed and a processing method for the data to be processed, and the processing method indicates a processing logic; Determine one or more common processing logics included in the processing method, where the common processing logic refers to scripts corresponding to the same logic part in the processing logic indicated by the processing method; According to the one or more common processing logics, obtaining a task template; Generate a data processing task according to the task template and the data to be processed; executing the data processing task in response to the data processing request; Among them, the data processing request also indicates the time parameter corresponding to the data to be processed; the task template includes: time parameter format; converting the time parameter into a target format corresponding to the time parameter format; according to the target format, obtaining the data to be processed corresponding to the time parameter from a database; the time parameter format includes an operator string, the operator string begins with the currently determined time parameter, includes multiple calculation operators in the middle, and the last operator is a format operator. Other operators after the first format operator are invalid. When the first format operator is current, it means that the previous calculation operator is invalid.

2. The method according to claim 1, characterized in that In the case where the plurality of common processing logics correspond to a plurality of the task templates, Determine whether there is a dependency relationship between the multiple task templates; If so, the data processing tasks corresponding to the multiple task templates are executed in sequence according to the dependency relationship.

3. The method according to claim 2, characterized in that The task template is configured with task dependency information; It is determined whether the plurality of task templates have the dependency relationship according to the task dependency information respectively included in the plurality of task templates.

4. The method according to claim 2, characterized in that: In the case where the plurality of task templates do not have the dependency relationship, The data processing tasks respectively corresponding to the multiple task templates are executed in parallel.

5. The method according to claim 1, characterized in that The step of generating a data processing task according to the task template and the data to be processed includes: Determine, according to the data to be processed, feature processing logic other than the common processing logic; The data processing task is generated according to the data to be processed corresponding to the feature processing logic, the data to be processed corresponding to the common processing logic, and the task template.

6. The method according to claim 5, characterized in that The generating the data processing task according to the data to be processed corresponding to the feature processing logic, the data to be processed corresponding to the common processing logic, and the task template includes: Determine a first script corresponding to the task template and the feature processing logic according to the feature processing logic, the data to be processed corresponding to the feature processing logic, the common processing logic corresponding to the task template, and the data to be processed corresponding to the task template; The data processing task is generated according to the first script.

7. The method according to claim 1, characterized in that Also includes: Get historical execution data of multiple historical tasks; Determining, based on the historical execution data, a common processing logic shared by the plurality of historical tasks; A second script corresponding to the common processing logic shared by the plurality of historical tasks is generated, and the task template is generated according to the second script.

8. The method according to claim 3, characterized in that The task template also includes: task description information, task parameter information, and task scheduling information; wherein the task parameter information includes any one or more of the following: task type, storage path, and execution thread.

9. The method according to claim 8, characterized in that The task parameter information includes: a data source identifier corresponding to the task template; The data source corresponding to the data source identifier is monitored, and when it is detected that the name of the data source has changed, the data source identifier in the task template is updated.

10. The method according to claim 8, characterized in that Different task types correspond to different task templates.

11. The method according to claim 1, characterized in that: Before generating the data processing task, the method further includes: Obtaining a configuration file, wherein the configuration file includes a system configuration file and an application configuration file; According to the configuration file, the to-be-processed data corresponding to the time parameter is acquired.

12. A data processing device, characterized in that: include: An acquisition module, used for acquiring a data processing request, wherein the data processing request indicates the data to be processed and a processing method of the data to be processed, and the processing method indicates a processing logic; A logic processing module, used to determine one or more common processing logics included in the processing method, wherein the common processing logic refers to a script corresponding to the same logic part in the processing logic indicated by the processing method; The acquisition module is further used to acquire the task template according to the one or more common processing logics; A task generation module, used to generate a data processing task according to the task template and the data to be processed; An execution module, used for executing the data processing task in response to the data processing request; Among them, the data processing request also indicates the time parameter corresponding to the data to be processed; the task template includes: time parameter format; converting the time parameter into a target format corresponding to the time parameter format; according to the target format, obtaining the data to be processed corresponding to the time parameter from a database; the time parameter format includes an operator string, the operator string begins with the currently determined time parameter, includes multiple calculation operators in the middle, and the last operator is a format operator. Other operators after the first format operator are invalid. When the first format operator is current, it means that the previous calculation operator is invalid.

13. A data processing electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images