Data processing method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a data processing method, device, electronic device and storage medium, the method is used in a big data processing system, the big data processing system includes: a data processing operator and an adapter, the adapter is used to perform corresponding packaging and adaptation of the data processing operator according to the interface standards of different computing platforms, the method includes: determining a data processing operator for generating a data processing task; using the adapter to perform packaging and adaptation on the data processing operator so that the packaged and adapted data processing operator meets the interface standard of the corresponding computing platform; generating a data processing task according to the packaged and adapted data processing operator; sending the data processing task to the computing platform so that the computing platform processes it. Based on the above implementation, the data processing operator can be sent to different computing platforms for processing.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the rapid development of Internet companies' business, data has also grown explosively. It is increasingly important to provide business departments with easy-to-use, stable and efficient real-time data services. Therefore, business operations based on real-time computing of big data have begun to emerge and are increasingly being used online, such as real-time recommendations, Double Eleven real-time large-screen statistics, real-time anti-fraud, etc.

[0003] Although there are many ways to implement big data processing, big data processing in the prior art is usually bound to a specific computing platform, which makes it inconvenient to use and not suitable for expansion. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a data processing method, device, electronic device and storage medium, which can realize the processing of big data on different platforms.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is used in a big data processing system, wherein the big data processing system includes: a data processing operator and an adapter, wherein the adapter is used to perform corresponding encapsulation adaptation on the data processing operator according to the interface standards of different computing platforms, and the method includes:

[0006] determining a data processing operator for generating a data processing task;

[0007] Using an adapter to encapsulate and adapt the data processing operator so that the encapsulated and adapted data processing operator complies with the interface standard of the corresponding computing platform;

[0008] Generate a data processing task according to the encapsulated and adapted data processing operator;

[0009] The data processing task is sent to the computing platform, so that the computing platform processes the data according to the data processing task.

[0010] In the above implementation process, first determine the data processing operator, which has the ability to process data. In order to process large-scale data, it is necessary to send the data processing operator to the computing platform so that the computing platform calls the data processing operator to process the data. Different computing platforms have different requirements for data processing operators. Therefore, use an adapter to encapsulate and adapt the data processing operator so that the encapsulated and adapted data processing operator meets the interface requirements of the computing platform for the data processing operator. Generate a data processing task based on the encapsulated and adapted data processing operator; send the data processing task to the computing platform for processing, so that the computing platform can call the data processing operator to process the data. Based on the above implementation method, the data processing operator can be sent to different computing platforms for processing, which improves the reusability and scalability of the data processing operator.

[0011] Furthermore, the big data processing system further comprises: a plug-in framework and a plug-in framework interface, wherein the plug-in framework interface has an interface standard;

[0012] The step of encapsulating and adapting the data processing operator by using an adapter includes:

[0013] The data processing operator is modified according to the interface standard of the plug-in framework interface to obtain a data processing plug-in;

[0014] The plug-in framework is driven to call the data processing plug-in through the plug-in framework interface, and the data processing plug-in is encapsulated and adapted through the adapter.

[0015] In the above implementation process, a plug-in framework and a plug-in framework interface are set. The plug-in framework interface has the corresponding plug-in framework interface standard. The data processing operator is transformed according to the interface standard to obtain a data processing plug-in. The plug-in framework can call the data processing plug-in through the interface and encapsulate and adapt the data plug-in through the adapter. Based on the above implementation method, automatic encapsulation and adaptation of the data processing operator can be achieved based on the framework, thereby improving the efficiency of encapsulation and adaptation.

[0016] Furthermore, the interface standard of the plug-in framework does not include the following constraints:

[0017] Constraints on the input data format and output data format of the data processing operator.

[0018] In the above implementation process, the interface standard does not include constraints on the input data type and output data type of the interface, thereby improving the flexibility of the data processing operator, reducing the coupling between the data processing operator and the plug-in framework, and improving the reusability of the data processing operator.

[0019] Furthermore, after encapsulating and adapting the data processing operator, the method further includes:

[0020] The driver plug-in framework obtains configuration parameters, and the configuration parameters are used to initialize the data processing operator.

[0021] In the above implementation process, the data processing operator is initialized by obtaining configuration parameters, so that the data processing operator can adapt to different data, further improving the reusability of the data processing operator.

[0022] Furthermore, generating a data processing task according to the encapsulated and adapted data processing operator includes:

[0023] Generate a directed acyclic graph with the data processing operator as a node;

[0024] The data processing task is generated according to the directed acyclic graph.

[0025] In the above implementation process, a directed acyclic graph is first generated, and then a data processing task is generated according to the directed acyclic graph, so that the computing platform can call the data processing operator according to the order in the directed acyclic graph to process the data, thereby accelerating the processing efficiency of the computing platform.

[0026] Furthermore, the big data processing system further comprises: a data processing engine, the data processing engine calling the data processing operator through the interface of the data processing operator;

[0027] The step of generating a directed acyclic graph with the data processing operator as a node includes:

[0028] Determining a connection order between data processing operators for generating data processing tasks;

[0029] Drive the data processing engine to generate the directed acyclic graph with each data processing operator as a node according to the connection order, so that the computing platform calls the data processing operator to process the data according to the connection order of the directed acyclic graph.

[0030] In the above implementation process, the data processing engine calls the data processing operator through the interface of the data processing operator, and generates a directed acyclic graph with each data processing operator as a node according to the connection order. This can avoid manual operation of each data processing operator in the process of generating the directed acyclic graph, thereby improving the efficiency of generating the directed acyclic graph.

[0031] Further, generating the data processing task according to the directed acyclic graph includes:

[0032] An input plug-in is added before the start node of the directed acyclic graph, and an output plug-in is added after the end node of the directed acyclic graph to obtain the data processing task, wherein the input plug-in is used to input the data into the directed acyclic graph, and the output plug-in is used to output the processed data to the directed acyclic graph.

[0033] In the above implementation process, an output plug-in and an input plug-in are set on the basis of the directed acyclic graph, the output plug-in is used to input data into the directed acyclic graph, and the output plug-in is used to output processed data. Based on the above implementation, the computing platform can directly process and output data through data processing tasks.

[0034] Further, after generating the data processing task according to the encapsulated and adapted data processing operator, the method further includes:

[0035] According to the directed acyclic graph, arbitrarily determine a first data processing operator and a second data processing operator connected along a connection sequence of the directed acyclic graph among the plurality of data processing operators;

[0036] Determining whether an output data type of the first data processing operator matches an input data type of the second data processing operator;

[0037] If not, a warning message is issued.

[0038] In the above implementation process, since the computing platform processes data according to the connection order of each node in the directed acyclic graph, and the output data type and input data type of each data processing operator are not exactly the same, therefore, when performing data calculation in the order in the directed acyclic graph, if the output data type and input type of the first data processing operator and the second data processing operator connected before and after do not match, an error will occur in the processing process of the computing platform. Therefore, before generating a data processing task, it is determined whether the output data type of the first data processing operator and the input data type of the second data processing operator match. If not, an alarm message is issued.

[0039] In a second aspect, an embodiment of the present application provides a data processing device, which is applied to a big data processing system, and the device includes:

[0040] A determination module, used to determine a data processing operator for generating a data processing task;

[0041] A packaging and adaptation module, used to use an adapter to package and adapt the data processing operator so that the packaged and adapted data processing operator complies with the interface standard of the corresponding computing platform;

[0042] A generation module, used to generate a data processing task according to the encapsulated and adapted data processing operator;

[0043] The sending module is used to send the data processing task to the computing platform so that the computing platform processes the data according to the data processing task.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.

[0046] Other features and advantages disclosed in the present application will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology disclosed in the present application.

[0047] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a process for encapsulating and adapting a data processing operator provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0054] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0055] Example 1

[0056] See also Figure 1 The embodiment of the present application provides a data processing method, which is used in a big data processing system. The big data processing system includes: a data processing operator and an adapter, and the adapter is used to perform corresponding packaging and adaptation of the data processing operator according to the interface standards of different computing platforms, including:

[0057] S1: Determine the data processing operator used to generate the data processing task;

[0058] S2: Encapsulate and adapt the data processing operator using an adapter so that the encapsulated and adapted data processing operator complies with the interface standard of the corresponding computing platform;

[0059] S3: Generate data processing tasks according to the encapsulated and adapted data processing operators;

[0060] S4: Send the data processing task to the computing platform so that the computing platform processes the data according to the data processing task.

[0061] In the above embodiment, the data processing operator defines a method for processing data. The computing platform can obtain data through a big data processing system or other methods. This embodiment does not limit the specific method of obtaining data by the computing platform.

[0062] For example, in an object-oriented programming language, data processing operators and adapters are in the form of objects, and the adapter is essentially a wrapper in the design pattern. By using the adapter, the data processing operator can be packaged so that the packaged data processing operator has certain functions, etc., thereby meeting the interface requirements of the computing platform for the data processing operator. It can be understood that different computing platforms correspond to different adapters based on different interface requirements of the computing platform, and the same computing platform has one or more adapters corresponding to the computing.

[0063] In the above implementation process, firstly, a data processing operator is determined. The data processing operator has the ability to process data. In order to process large-scale data, the data processing operator needs to be sent to a computing platform for processing. Different computing platforms have different requirements for data processing operators. Therefore, an adapter is used to encapsulate and adapt the data processing operator so that the encapsulated and adapted data processing operator meets the interface requirements of the computing platform for the data processing operator, so that the computing platform can call the data processing operator to process the data. Then, a data processing task is generated according to the encapsulated and adapted data processing operator, and the data processing task is sent to the computing platform for processing. Based on the above implementation method, the data processing operator can be sent to different computing platforms for processing.

[0064] Based on the above implementation method, data processing tasks can be embedded in common distributed computing frameworks, such as Flink, batch computing frameworks, such as Spark, and even in some data access systems, such as Flume, to achieve data access and complete processing at the same time.

[0065] See also Figure 2 In a possible implementation, the big data processing system further includes: a plug-in framework and a plug-in framework interface, the plug-in framework interface having an interface standard; S2 includes:

[0066] S21: transform the data processing operator according to the interface standard of the plug-in framework interface to obtain a data processing plug-in;

[0067] S22: The driver plug-in framework calls the data processing plug-in through the plug-in framework interface, and encapsulates and adapts the data processing plug-in through the adapter.

[0068] S21 includes defining the output data type and input data type of the data processing operator.

[0069] For example, the interface standard may include constraints on the output data type and the output data type, or need to include a method with a specific name. When a data processing operator is written in an object-oriented programming language, it is also necessary to indicate that the data processing operator complies with the interface standard of a certain interface according to the format of the object-oriented programming language during modification.

[0070] In the above implementation process, a plug-in framework and a plug-in framework interface are set. The plug-in framework interface has the corresponding plug-in framework interface standard. The data processing operator is transformed according to the interface standard to obtain a data processing plug-in. The plug-in framework can call the data processing plug-in through the interface and encapsulate and adapt the data plug-in through the adapter. Based on the above implementation method, automatic encapsulation and adaptation of the data processing operator can be achieved based on the framework, thereby improving the efficiency of encapsulation and adaptation.

[0071] In a possible implementation, the interface standard of the plug-in framework does not include the following constraints:

[0072] Constraints on the input and output data formats of data processing operators.

[0073] For example, when an object-oriented programming language is used to define a plug-in framework interface, generic methods and the like may be used to define the plug-in framework interface and methods within the plug-in framework interface, thereby avoiding constraints on output types and input types of data processing operators.

[0074] In the above implementation process, the interface standard does not include constraints on the input data type and output data type of the interface, thereby improving the flexibility of the data processing operator, reducing the coupling between the data processing operator and the plug-in framework, and improving the reusability of the data processing operator.

[0075] In a possible implementation, after encapsulating and adapting the data processing operator, the method further includes:

[0076] The driver plug-in framework obtains configuration parameters, which are used to initialize the data processing operator.

[0077] In the above implementation process, the data processing operator is initialized by obtaining configuration parameters, so that the data processing operator can adapt to different data, further improving the reusability of the data processing operator.

[0078] In a possible implementation, an initialization module for initializing the data processing operator is provided in the data processing operator, and the plug-in framework initializes the data processing operator by calling the module of the data processing operator.

[0079] Exemplarily, if the data processing operator is implemented in an object-oriented programming language, the corresponding initialization module may be a function of configuration parameters of the data processing operator, etc. Based on the initialization module, the performance of the data processing operator may be adjusted.

[0080] In a possible implementation, S3 includes: generating a directed acyclic graph with data processing operators as nodes; and generating a data processing task according to the directed acyclic graph.

[0081] In the above implementation process, a directed acyclic graph is first generated, and then a data processing task is generated according to the directed acyclic graph, so that the computing platform can call the data processing operator to process the data according to the connection order in the directed acyclic graph, thereby accelerating the processing efficiency of the computing platform.

[0082] In a possible implementation, the big data processing system further includes: a data processing engine, the data processing engine calling the data processing operator through the interface of the data processing operator;

[0083] Generating a directed acyclic graph with data processing operators as nodes includes:

[0084] Determining a connection order between data processing operators for generating data processing tasks;

[0085] Drive the data processing engine to generate a directed acyclic graph with each data processing operator as a node according to the connection order, so that the computing platform calls the data processing operator to process the data according to the connection order of the directed acyclic graph.

[0086] In the above implementation process, the data processing engine calls the data processing operator through the interface of the data processing operator, and generates a directed acyclic graph with each data processing operator as a node according to the connection order. This can avoid manual operation of each data processing operator in the process of generating the directed acyclic graph, thereby improving the efficiency of generating the directed acyclic graph.

[0087] In a possible implementation, generating a data processing task according to a directed acyclic graph includes:

[0088] An input plug-in is added before the start node of the directed acyclic graph, and an output plug-in is added after the end node of the directed acyclic graph to obtain a data processing task. The data input plug-in is used to input data into the directed acyclic graph, and the data output plug-in is used to output the processed data.

[0089] In the above implementation process, an output plug-in and an input plug-in are added on the basis of the directed acyclic graph, the output plug-in is used to input data into the directed acyclic graph, and the data output plug-in is used to output the processed data. Based on the above implementation, the computing platform can output the data after computing through the data processing task.

[0090] When the interface standard of the plugin interface does not include constraints on the output data type and input data type of the data processing operator, after S3, it also includes:

[0091] According to the directed acyclic graph, arbitrarily determine a first data processing operator and a second data processing operator connected along a connection order of the directed acyclic graph among a plurality of data processing operators;

[0092] Determine whether the output data type of the first data processing operator matches the input data type of the second data processing operator;

[0093] If not, a warning message is issued.

[0094] In the above implementation process, since the computing platform processes data according to the connection order of each node in the directed acyclic graph, and the output data type and input data type of each data processing operator are not exactly the same, therefore, when performing data calculation in the order in the directed acyclic graph, if the output data type and input type of the first data processing operator and the second data processing operator connected before and after do not match, an error will occur in the processing process of the computing platform. Therefore, before generating a data processing task, it is determined whether the output data type of the first data processing operator and the input data type of the second data processing operator match. If not, an alarm message is issued.

[0095] Example 2

[0096] See also Figure 3 The present application embodiment provides a data processing device, which is applied to a big data processing system, and the device includes:

[0097] Determining module 1, used to determine a data processing operator for generating a data processing task;

[0098] The encapsulation and adaptation module 2 is used to encapsulate and adapt the data processing operator using the adapter so that the encapsulated and adapted data processing operator complies with the interface standard of the corresponding computing platform;

[0099] A generation module 3 is used to generate a data processing task according to the encapsulated and adapted data processing operator;

[0100] The sending module 4 is used to send the data processing task to the computing platform so that the computing platform processes the data according to the data processing task.

[0101] In the above implementation process, first determine the data processing operator, which has the ability to process data. In order to process large-scale data, the data processing operator needs to be sent to the computing platform for processing. Different computing platforms have different requirements for data processing operators. Therefore, use an adapter to encapsulate and adapt the data processing operator so that the encapsulated and adapted data processing operator meets the interface requirements of the computing platform for the data processing operator, so that the computing platform can call the data processing operator to process the data. Then, generate a data processing task based on the encapsulated and adapted data processing operator, and send the data processing task to the computing platform for processing. Based on the above implementation, the data processing operator can be sent to different computing platforms for processing.

[0102] In one possible implementation, the big data processing system also includes: a plug-in framework and a plug-in framework interface, the plug-in framework interface has an interface standard; the encapsulation adaptation module 2 is also used to transform the data processing operator according to the interface standard of the plug-in framework interface to obtain a data processing plug-in; the driving plug-in framework calls the data processing plug-in through the plug-in framework interface, and encapsulates and adapts the data processing plug-in through the adapter.

[0103] In a possible implementation, the interface standard of the plug-in framework does not include the following constraints: constraints on the input data format and output data format of the data processing operator.

[0104] In a possible implementation, the device further includes a configuration module, which is used to drive the plug-in framework to obtain configuration parameters, and the configuration parameters are used to initialize the data processing operator.

[0105] In a possible implementation, the generation module 3 is further used to generate a directed acyclic graph with data processing operators as nodes; and generate a data processing task according to the directed acyclic graph.

[0106] In one possible implementation, the big data processing system also includes: a data processing engine, which calls the data processing operator through the interface of the data processing operator, and the generation module 3 is also used to determine the connection order between the data processing operators used to generate the data processing tasks; driving the data processing engine, generating a directed acyclic graph with each data processing operator as a node according to the connection order, so that the computing platform calls the data processing operator according to the connection order of the directed acyclic graph to process the data.

[0107] In a possible implementation, the generation module 3 is also used to add an input plug-in before the starting node of the directed acyclic graph and to add an output plug-in after the ending node of the directed acyclic graph to obtain a data processing task. The data input plug-in is used to input data into the directed acyclic graph, and the data output plug-in is used to output processed data.

[0108] In a possible implementation, the device also includes a checking module for arbitrarily determining, according to the directed acyclic graph, a first data processing operator and a second data processing operator connected in a connection order of the directed acyclic graph among multiple data processing operators; judging whether the output data type of the first data processing operator matches the input data type of the second data processing operator; and if not, issuing an alarm message.

[0109] Example 3

[0110] This application also provides an electronic device, see Figure 4 , Figure 4A block diagram of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 41, a communication interface 42, a memory 43, and at least one communication bus 44. The communication bus 44 is used to realize direct connection and communication between these components. The communication interface 42 of the electronic device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor 41 may be an integrated circuit chip with signal processing capabilities.

[0111] The processor 41 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor 41 can also be any conventional processor, etc.

[0112] The memory 43 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 43 stores computer-readable instructions, and when the computer-readable instructions are executed by the processor 41, the electronic device may execute the various steps involved in the above method embodiment.

[0113] Optionally, the electronic device may further include a storage controller and an input / output unit.

[0114] The memory 43, storage controller, processor 41, peripheral interface, input and output unit components are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 44. The processor 41 is used to execute executable modules stored in the memory 43, such as software function modules or computer programs included in the electronic device.

[0115] The input and output unit is used to provide users with the task creation and to create an optional time period or preset execution time for the task to enable interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.

[0116] Understandably, Figure 4 The structure shown is for illustration only. The electronic device may also include Figure 4 More or fewer components as shown, or with Figure 3 Different configurations are shown. Figure 4 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0117] The embodiment of the present application also provides a computer-readable storage medium, on which instructions are stored. When the instructions are run on a computer, the method of the method embodiment is implemented when the computer program is executed by the processor. To avoid repetition, it will not be described here.

[0118] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0120] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0121] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0123] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A data processing method, It is characterized in that The method is used in a big data processing system, the big data processing system comprising: a data processing operator and an adapter, the adapter being used to perform corresponding encapsulation adaptation on the data processing operator according to the interface standards of different computing platforms, the method comprising: determining a data processing operator for generating a data processing task; Using an adapter to encapsulate and adapt the data processing operator so that the encapsulated and adapted data processing operator complies with the interface standard of the corresponding computing platform; Generate a data processing task according to the encapsulated and adapted data processing operator; Sending the data processing task to the computing platform so that the computing platform processes the data according to the data processing task; The big data processing system further includes: a plug-in framework and a plug-in framework interface, wherein the plug-in framework interface has an interface standard; the plug-in framework interface standard does not include the following constraints: constraints on the input data format and output data format of the data processing operator; The step of encapsulating and adapting the data processing operator by using an adapter includes: The data processing operator is modified according to the interface standard of the plug-in framework interface to obtain a data processing plug-in; The plug-in framework is driven to call the data processing plug-in through the plug-in framework interface, and the data processing plug-in is encapsulated and adapted through the adapter.

2. The data processing method according to claim 1, It is characterized in that After encapsulating and adapting the data processing operator, the method further includes: The driver plug-in framework obtains configuration parameters, and the configuration parameters are used to initialize the data processing operator.

3. The data processing method according to claim 2, It is characterized in that Generating a data processing task according to the encapsulated and adapted data processing operator includes: Generate a directed acyclic graph with the data processing operator as a node; The data processing task is generated according to the directed acyclic graph.

4. The data processing method according to claim 3, It is characterized in that The big data processing system further includes: a data processing engine, the data processing engine calling the data processing operator through the interface of the data processing operator; The step of generating a directed acyclic graph with the data processing operator as a node includes: Determining a connection order between data processing operators for generating data processing tasks; Drive the data processing engine to generate the directed acyclic graph with each data processing operator as a node according to the connection order, so that the computing platform calls the data processing operator to process the data according to the connection order of the directed acyclic graph.

5. The data processing method according to claim 3, It is characterized in that Generating the data processing task according to the directed acyclic graph includes: An input plug-in is added before the start node of the directed acyclic graph, and an output plug-in is added after the end node of the directed acyclic graph to obtain the data processing task, wherein the input plug-in is used to input the data into the directed acyclic graph, and the output plug-in is used to output the processed data to the directed acyclic graph.

6. The data processing method according to claim 3, It is characterized in that After generating the data processing task according to the encapsulated and adapted data processing operator, the method further includes: According to the directed acyclic graph, arbitrarily determine a first data processing operator and a second data processing operator connected along a connection sequence of the directed acyclic graph among the plurality of data processing operators; Determining whether an output data type of the first data processing operator matches an input data type of the second data processing operator; If not, a warning message is issued.

7. A data processing device, It is characterized in that Applied to a big data processing system, the device comprises: A determination module, used to determine a data processing operator for generating a data processing task; A packaging and adaptation module, used to use an adapter to package and adapt the data processing operator so that the packaged and adapted data processing operator complies with the interface standard of the corresponding computing platform; A generation module, used to generate a data processing task according to the encapsulated and adapted data processing operator; A sending module, used for sending the data processing task to the computing platform, so that the computing platform processes the data according to the data processing task; The big data processing system further includes: a plug-in framework and a plug-in framework interface, the plug-in framework interface having an interface standard; the plug-in framework interface standard does not include the following constraints: constraints on the input data format and output data format of the data processing operator; The encapsulation adaptation module is also used to transform the data processing operator according to the interface standard of the plug-in framework interface to obtain the data processing plug-in; the driver plug-in framework calls the data processing plug-in through the plug-in framework interface, and encapsulates and adapts the data processing plug-in through the adapter.

8. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to execute the steps of the data processing method according to any one of claims 1 to 6.

Images