Data processing methods, apparatus, electronic devices and storage media

By writing task processing requests using a predefined interface description language, the problem of inconsistent interfaces between different data processing engines is solved, enabling unified cross-system calls, reducing learning costs, and improving the system's versatility and performance.

CN115857918BActive Publication Date: 2026-11-10NETEASE (HANGZHOU) NETWORK CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202211690214.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-27
Publication Date
2026-11-10
Estimated Expiration
2042-12-27

AI Technical Summary

Technical Problem

Existing data processing systems suffer from incompatible application interface languages ​​for different processing engines, requiring developers to learn multiple specialized languages, which increases learning costs and raises the barrier to entry.

Method used

Task processing requests are written using a predefined interface description language. The target processing system is determined by parsing key fields, and the task processing request is converted into the execution code of the target processing system to achieve unified cross-system invocation.

Benefits of technology

It reduces the learning cost for users, improves the versatility and performance of the data processing system, provides a unified data access interface, supports multiple data models and computation operators, and integrates storage and computing capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115857918B_ABST
    Figure CN115857918B_ABST
Patent Text Reader

Abstract

This application provides a data processing method, apparatus, electronic device, and storage medium, relating to the field of data processing technology. The method involves writing task processing requests using a predefined interface description language. By parsing key fields in the task processing request, the target processing system corresponding to the request can be determined. The task processing request is then converted into executable code supported by the target processing system. The target processing system runs the executable code to execute the task processing request and generates the task execution result. This method, by using a predefined interface description language to write task processing requests, allows different users to call various target processing systems within the data processing system simply by learning the unified interface description language. This significantly reduces the learning cost for users, lowers the learning threshold of the data processing system, and improves its versatility and performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Data storage and data computing are important technologies in the development of data processing. Powerful data processing systems all operate based on data storage and data computing systems to meet various data processing needs.

[0003] Data storage systems and data computing systems typically include multiple processing engines to comprehensively adapt to data computing in various scenarios. However, because different processing engines use different application interface languages, developers need to learn the specific languages ​​of each processing engine, resulting in a high learning cost for developers. Summary of the Invention

[0004] The purpose of this application is to address the shortcomings of the prior art by providing a data processing method, apparatus, electronic device, and storage medium, so as to solve the problems of high barriers to entry and high learning costs for users in the existing data processing systems.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0006] In a first aspect, embodiments of this application provide a data processing method applied to a data processing system, the method comprising:

[0007] The task processing request is obtained as input. The task processing request includes: data to be processed and operations to be performed. The task processing request is written in a predefined interface description language.

[0008] Based on the task processing request, determine the target processing system corresponding to the task processing request;

[0009] The task processing request is converted into executable code corresponding to the target processing system and submitted to the target processing system. The executable code is then run by the target processing system to generate the task execution result corresponding to the task processing request.

[0010] Secondly, embodiments of this application also provide a data processing apparatus for use in a data processing system, the apparatus comprising: an acquisition module, a determination module, and an execution module;

[0011] The acquisition module is used to acquire the input task processing request, which includes: data to be processed and operations to be performed. The task processing request is written in a predefined interface description language.

[0012] The determining module is used to determine the target processing system corresponding to the task processing request based on the task processing request.

[0013] The execution module is used to convert the task processing request into execution code corresponding to the target processing system, submit it to the target processing system, and run the execution code through the target processing system to generate the task execution result corresponding to the task processing request.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the data processing method provided in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the data processing method provided in the first aspect.

[0016] The beneficial effects of this application are:

[0017] This application provides a data processing method, apparatus, electronic device, and storage medium. Task processing requests are written using a predefined interface description language. By parsing key fields in the task processing request, the target processing system corresponding to the request can be determined. The task processing request is then converted into executable code supported by the target processing system. The target processing system runs the executable code to execute the task processing request and generate the task execution result. This method, by using a predefined interface description language to write task processing requests, allows different users to call various target processing systems within the data processing system simply by learning the unified interface description language. This significantly reduces the learning cost for users, lowers the learning threshold of the data processing system, and improves its versatility and performance.

[0018] The data capability system, implemented based on the defined interface description language, integrates data storage and computing capabilities, providing a unified perspective for operation, development, and use, achieving a good balance between performance and versatility. It supports more data models and provides clearer data access interfaces; it merges computing and storage systems, eliminating the need to design complex connectors for various storage systems. This data processing system simultaneously supports batch and streaming data processing and a rich set of computational operators, enabling users to easily expand and develop the data processing system based on existing operators. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 Flowchart of the data processing method provided in the embodiments of this application Figure 1 ;

[0021] Figure 2 Flowchart of the data processing method provided in the embodiments of this application Figure 2 ;

[0022] Figure 3 This application provides a schematic diagram of the architecture of a data processing system.

[0023] Figure 4 This is a schematic diagram of the architecture of another data processing system provided in an embodiment of this application;

[0024] Figure 5 Flowchart of the data processing method provided in the embodiments of this application Figure 3 ;

[0025] Figure 6 Flowchart of the data processing method provided in the embodiments of this application Figure 4 ;

[0026] Figure 7 A schematic diagram of a data processing apparatus provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0029] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0031] First, a brief explanation of the relevant background information for this plan:

[0032] Data technologies can be categorized by system type (e.g., databases, big data systems), by data processing model (e.g., relational databases, document databases, search engines, graph databases), and by processing timeliness (e.g., real-time data processing, offline data processing). Around these data technologies, a wide variety of software has emerged, such as MySQL, PostgreSQL, and Oracle in the database field, and Hadoop, Hive, Spark, and Flink in the big data field. These software programs are built on top of operating systems, are widely compatible with various hardware, and have evolved from single-machine systems to distributed systems over the past few decades.

[0033] While complex data software solves many problems in the business domain, it also introduces complexity. For example, building a big data system based on Hadoop involves deploying more than a dozen key components (Hadoop MapReduce, Yarn, Hive, Kafka, Elasticsearch, Spark, Flink, etc.). These components differ not only in deployment complexity but also in usage and long-term maintenance difficulty, placing a heavy burden on developers. For instance, data developers responsible for business log ingestion not only need to write log parsing logic using the programming APIs (Application Programming Interfaces) provided by computing frameworks (MR (MapReduce, a distributed computing programming framework), Hive, Spark, Flink, etc.), but also need to be familiar with the use of upstream and downstream components, such as how to read data from message queues and how to write parsed data to various downstream storage engines.

[0034] The complex and diverse data technologies not only introduce significant learning costs for professional data developers, but also create a high barrier to entry for those engaged in algorithm development, data science, and related fields. For example, algorithm engineers typically need to read data in batches from various big data storage systems, perform certain cleaning and transformation processes to build training and testing sets, and then train machine learning / deep learning models. However, they often encounter many usage problems during the data reading, cleaning, and transformation process, such as being unable to find a suitable software development kit (SDK) for their own runtime environment, or the SDK being difficult to use, forcing them to delve into the technical details related to big data.

[0035] Currently, SQL (Structured Query Language) is a widely used domain-specific programming language (DSL) for data manipulation in the database field. Traditional relational databases (MySQL / Oracle, etc.) and big data computing frameworks (Spark / Flink, etc.) all support writing data processing logic using SQL. They convert user-input SQL into corresponding execution plans through a parser and send them to the executor for execution, realizing data CRUD operations.

[0036] SQL has brought immense convenience to data-related work, especially structured data. Whether you're a data developer, data analyst, or data scientist, you can access data using a unified SQL language without needing to understand the underlying system details. Different database engines (such as MySQL, SQL Server, PostgreSQL, etc.) significantly lower the learning curve by implementing SQL interfaces, eliminating the need for users to learn a specific DSL language for any particular database system.

[0037] However, with the maturity of big data storage technologies, many distributed NoSQL databases that support large-scale data volumes do not support data manipulation via SQL (such as Cassandra and HBase). These systems typically implement their own data access API protocols and query languages. As software systems become older and more complex, software developers have to learn how to operate different databases, resulting in significant learning costs.

[0038] Beyond storage technologies, an increasing number of big data computing engines (such as Spark and Flink) are also using SQL as an API for the user interface. Users write data processing code in SQL (such as SparkSQL), which is then submitted to the big data computing engine for translation and execution. Although storage and computing systems use similar SQL languages, their levels of support and the primitives they use are not entirely the same. For example, in big data streaming computing scenarios, significant extensions have been made to the original SQL, resulting in a language that is not entirely identical to the SQL used in storage systems.

[0039] Whether it's a storage system or a computing system, they are usually tightly bound to certain software and hardware systems. For example, some databases can only run on specific operating systems and hardware devices (such as IBM's DB2). Due to the limitations of the software and hardware environment, developers often need to try to migrate the original system to the new system and relearn the operation and use of the new system, which brings a considerable burden to the developers. Meanwhile, the underlying database systems, due to the constraints of the ecosystem, are often unwilling to migrate their systems to other software and hardware platforms, further resulting in a situation where the entire storage and computing ecosystem is fragmented and filled with products.

[0040] Therefore, there is an urgent need for a more universal interface description language that can unify access to various storage systems, unify the use of various data computing frameworks, and be compatible with various software and hardware systems. Systems that implement a universal interface description language, whether storage or computing systems, can use a unified interface description language to describe the semantics of data storage and computation; both data developers and data scientists can use a unified interface description language to describe data processing logic, thereby greatly reducing the barrier to entry for using data processing systems.

[0041] Based on this requirement, this solution proposes the following approach:

[0042] First, a general interface description language is defined, in which the core key fields of data processing are defined, which is the top-level abstraction of data processing.

[0043] Secondly, a data capability system was implemented based on the defined general interface description language. A series of basic interfaces were defined, and the system was adapted to various storage and computing frameworks through component-based design.

[0044] The implementation steps and principles of this solution are explained below through specific embodiments:

[0045] Figure 1 Flowchart of the data processing method provided in the embodiments of this application Figure 1 This method can be applied to data processing systems, which can be complete processing systems that include storage and computing systems. These systems can perform database access operations, such as adding, deleting, modifying, and querying data, as well as advanced operations, and can also perform complex data logic calculations, such as real-time data calculations and logical calculations.

[0046] like Figure 1 As shown, the method may include:

[0047] S101. Obtain the input task processing request. The task processing request includes: data to be processed and operations to be performed. The task processing request is written using a predefined interface description language.

[0048] Typically, when using a data processing system to perform data processing tasks, users can inform the system of their current data processing needs by sending task processing requests.

[0049] Optionally, the task processing request may include at least two parts: one part may be the data to be processed, that is, indicating what the data to be processed is, and the other part may be the operation to be performed, that is, indicating what data operation to be performed on the data to be processed.

[0050] In this embodiment, task processing requests can be written using a predefined interface description language. In this way, different users only need to learn a predefined set of interface description languages ​​to implement different data processing businesses. They do not need to master the dedicated interface languages ​​of each computing engine, database and component in the data processing system. This can effectively reduce the learning workload of users, lower the threshold for using the data processing system, and improve the performance and versatility of the data processing system.

[0051] S102. Based on the task processing request, determine the target processing system corresponding to the task processing request.

[0052] The target processing system here can refer to a specific subsystem within a data processing system. Target processing systems can be broadly categorized into storage systems and computing systems, each of which contains many subsystems. In this embodiment, the target processing system refers to one or more of these subsystems.

[0053] It can parse the key fields in the task processing request to determine the target processing system corresponding to the task processing request. The determined target processing system is used to process the processing task indicated by the task processing request.

[0054] S103. Convert the task processing request into the execution code corresponding to the target processing system, submit it to the target processing system, and run the execution code through the target processing system to generate the task execution result corresponding to the task processing request.

[0055] Since the task processing request is written in a predefined interface description language, which is not a language that the target processing system can recognize and execute, the task processing request can be converted into the corresponding execution code of the target processing system. The converted execution code can then be submitted to the target processing system, whereby the execution code will run to perform the processing requests made in the task processing request and generate the task execution result.

[0056] In some embodiments, the target processing system here does not refer to a single system. When the processing request requested by the task processing request is complex, it may be necessary to cooperate between multiple processing systems in the data processing system to execute the processing request. In this case, there may be multiple target processing systems, which cooperate with each other to execute the task processing request and finally generate the task execution result.

[0057] In summary, the data processing method provided in this embodiment uses a predefined interface description language to write task processing requests. By parsing the key fields in the task processing request, the target processing system corresponding to the request can be determined. The task processing request is then converted into executable code supported by the target processing system. The target processing system runs the executable code to execute the task processing request and generates the task execution result. This method, by using a predefined interface description language to write task processing requests, allows different users to call various target processing systems within the data processing system simply by learning the unified interface description language. This significantly reduces the learning cost for users, lowers the learning threshold of the data processing system, and improves its versatility and usability.

[0058] Figure 2 Flowchart of the data processing method provided in the embodiments of this application Figure 2 Optionally, in step S101, before obtaining the input task processing request, the following may be included:

[0059] S201. Create language description key fields using a preset language format. The language description key fields include: namespace, dataset, process, and task.

[0060] Optionally, a language definition with a preset format can be used to adapt to the interface description language of each target processing system in the data processing system. Here, key fields for language description can be created. These key fields can be the core keywords of data processing and represent the top-level abstraction of data processing.

[0061] Optionally, the language description key fields include data storage abstraction and data computation abstraction, which further abstract concepts such as namespace, dataset, process, and task.

[0062] In the task processing request, the data to be processed corresponds to the namespace key fields and the dataset key fields. The data to be processed can be determined by the namespace key fields and the dataset key fields.

[0063] Of course, there is also a case where the data to be processed can be specified directly using other fields. In this case, the data to be processed can be a specific number.

[0064] The operations to be executed in the task processing request can correspond to namespace key fields, process key fields, and task key fields. By using the namespace, process, and task key fields, it can be determined which specific task under which process will be called to execute the task request.

[0065] In this embodiment, the preset format language can be JSON. JSON is a lightweight data interchange format whose content consists of attributes (keys) and values. Due to its universality and openness, JSON has become the most commonly used data interchange format in the field of software development such as web development.

[0066] Of course, it is not limited to JSON; it can also be Python. When different preset formats are selected, some components of the data processing system will make adaptive adjustments.

[0067] S202. Generate the interface description language based on the key fields described in each language.

[0068] Optionally, based on the language description key fields created above, an interface description language can be generated, so that users can write task processing requests according to the various language description key fields defined in the interface description language.

[0069] Figure 3 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application. The data processing system may include: a data access abstraction and a data computation abstraction. The data access abstraction is used to express the addition, deletion, modification, and querying of data in the storage system, as well as high-level complex queries and processing. The data computation abstraction is used to express offline or real-time computing tasks in the computing system.

[0070] Whether it's data access abstraction or data computation abstraction, the top layer defines key fields in the namespace. The namespace is a logical workspace that enables unified management of resources such as datasets, metadata, and permissions. For example, the same namespace can include several datasets and several data processing flows.

[0071] For storage systems, the next level of the namespace defines the dataset key field. A dataset is the basic unit for managing data access. Each dataset is bound to a unique data source (such as a relational database or distributed file storage). A dataset stores several data sets with the same data model. For example, a dataset in a relational database stores several records with the same data model (schema).

[0072] For computing systems, the next level of the namespace also defines process key fields and task key fields.

[0073] A process is the basic unit for managing data computing tasks. Each process is bound to a unique computing scheduling engine (such as the Airflow scheduler, Yarn scheduler, etc.). A process contains several tasks, such as Spark computing tasks, Hive computing tasks, etc. Processes usually have business attributes. A series of data computing tasks that are closely related in terms of business are usually placed under the same process and form a hierarchical dependency relationship.

[0074] A task is the basic unit for performing data computation. Each task is bound to a unique operator (such as Spark operators, Flink operators, Hive operators, HTTP operators, etc.). Each task must belong to a certain process. Tasks are usually divided into short-running tasks and long-running tasks.

[0075] In addition to the basic key fields mentioned above, data access abstraction and data computation abstraction also include concepts such as metadata and authorization. For example, each dataset is bound to a set of metadata, which describes the basic model of the dataset. Another example is that namespaces are bound to several permissions, which control the user's access rights to datasets, processes, and tasks under the namespace.

[0076] Optionally, the namespace includes: a name field and a version field; the dataset includes: a name field and a source field; the process includes: a name field and a description field of the scheduler corresponding to the process; and the task includes: a name field and an operator field to which the task is bound.

[0077] The following section will use illustrative code to explain the specific fields included in each key field:

[0078] For namespaces, in addition to the basic name field, fields such as description (desc) and version (version) are also defined. The description field is optional, while the version field is required.

[0079] 1.{

[0080] 2. "name":"my_namespace",

[0081] 3. "desc":"This is a namespace",

[0082] 4. "version"1

[0083] }

[0084] Different versions of the namespace store different data, meaning the data in the namespace is constantly being updated and there will be multiple versions. Usually, when no specific version is specified, the latest version of the namespace can be selected by default.

[0085] In addition to the basic name field, the dataset also includes information such as description and version. The dataset also includes a source field, which is the same as the engine field shown in the code below. The source field and the engine field have the same meaning, representing the underlying storage engine of this dataset. The source field is a physical name for the source of the dataset, while the engine field is a logical name. In the following detailed description, the source field will be used as the source field of the dataset.

[0086] 1.{

[0087] 2."name":"ds1",

[0088] 3. "desc":"This is a dataset",

[0089] 4."version":1,

[0090] 5. "engine""mysql",

[0091] 6. "input":"task1"

[0092] }

[0093] "input":"task1", ​​"engine","mysql" indicates that the dataset named ds1 was generated by task 1 and stored in the mysql database. The task represents a data computation task.

[0094] For processes, their basic structure is similar to namespaces, including fields such as name, description, and version. In addition, processes also include a scheduler field, which indicates which scheduling engine (Airflow, Yarn, etc.) is used for the process.

[0095] 1.{

[0096] 2. "name":"workflow1",

[0097] 3. "desc":"This is a process".

[0098] 4. "version"1,

[0099] 5. "scheduler": "Airflow"

[0100] 6.}

[0101] Different schedulers are typically suited for different scenarios. For example, Airflow is suitable for offline batch processing workflows, while Yarn is suitable for big data offline + real-time hybrid job workflows.

[0102] A process can include a series of tasks. In addition to basic fields such as name, description, and version, each task also includes an operator field.

[0103] 1.{

[0104] 2. "name":"task1",

[0105] 3. "desc":"This is a task",

[0106] 4."version":1,

[0107] 5. "operator": "Spark"

[0108] 6.}

[0109] Here, the name of a task is usually unique within the same namespace and workflow. Therefore, it can be referenced between different datasets within the same workflow (source field). Datasets in different workflows can also form dependencies. However, in this case, the source field of the dataset needs to include the corresponding workflow name prefix. Datasets in different namespaces can also form dependencies. However, in this case, the source field of the dataset needs to include the namespace and workflow prefix.

[0110] Optionally, in step S101, obtaining the input task processing request may include: inputting the information corresponding to each field in the language description key fields defined in the interface description language, and generating a task processing request.

[0111] In some embodiments, users can describe each field in the key fields according to the defined language, input the corresponding task-related information in each field, and generate a task processing request. After writing and generating the task processing request, users can send the task processing request out through operations such as submission, so that the data processing system can receive and obtain the task processing request input by the user.

[0112] Optionally, in step S102, determining the target processing system corresponding to the task processing request based on the task processing request may include: determining the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

[0113] In some embodiments, the target processing system corresponding to a task processing request can be determined based on the source field of the dataset and / or the description field of the scheduler corresponding to the process in the task processing request. Since the dataset is a key language description field defined in the storage system of the data processing system, the source field of the dataset can determine that the task processing request invokes the storage system within the data processing system, and specifically, which storage engine within the storage system is invoked. Similarly, the process is a key language description field defined in the computing system of the data processing system, and the description field of the scheduler corresponding to the process can uniquely determine which computing engine within the computing system is invoked by the task processing request.

[0114] Therefore, based on the source field of the dataset and / or the description field of the scheduler corresponding to the process, the target processing system corresponding to the task processing request can be accurately determined.

[0115] Figure 4 This is a schematic diagram of the architecture of another data processing system provided in an embodiment of this application. Figure 4 As shown, the data processing system provided in this embodiment may include: an interpreter and at least one target processing system.

[0116] After determining the target processing system based on the source field of the dataset and the description field of the scheduler corresponding to the process, the next step is to ensure that the corresponding storage engine, scheduler, and operators in the target processing system can be invoked through task processing requests. This involves the translation problem from the defined interface description language to the target processing system. Based on this, an interpreter is also designed in the data processing system of this solution. The interpreter can translate the interface description language input by the user into executable code that the target processing system can run, so that the task processing request can be executed in the target processing system.

[0117] In one possible implementation, the interpreter may include: 1. a register; 2. a connector; 3. a generator; 4. an executor; and 5. a collector.

[0118] The registry is designed for different heterogeneous storage and computing systems. All storage engines, schedulers, and operators supported by the defined interface description language need to be registered in the registry. The registry is based on a strongly consistent central storage (such as ETCD) to ensure the atomicity of the registration process.

[0119] Connectors are used to connect different storage and computing engines. For example, for a storage engine, it is usually necessary to initialize the corresponding connection pool instance, and for a computing engine, it is usually necessary to initialize the corresponding computing session.

[0120] The code generator is the core component of the interpreter. It is used to generate the executable code of the corresponding target processing system based on the task processing request written in a predefined interface description language. The difficulty here lies in how to unify different storage engines or computing engines.

[0121] Based on this, optionally, a series of interfaces can be defined in the code generator, such as data CRUD interfaces, advanced interfaces, data calculation interfaces, etc. For different storage and computing engines, the corresponding interfaces need to be implemented.

[0122] For storage engines, the basic interfaces may include: Create (create), Delete (delete), Update (update), and Select (filter), which represent adding, deleting, modifying, and querying data, respectively.

[0123] The storage engine also includes a series of high-level interfaces, such as batch_create / delete / update / select, aggregate, group, order, count, slice, and limit.

[0124] For a computing engine, basic interfaces may include: map and reduce; these represent two typical stages of big data processing. The map stage divides the data into shards, and the mapper executes the corresponding map operation (map can be understood as a function) on each shard. Different shards are typically processed on different worker nodes, and the results are output as (key, value) tuples. After the map stage, the reduce stage begins. In the reduce stage, tuples with the same key are assigned to the same reducer for further aggregation. Between the map and reduce stages, the following stages may also be included: shuffle: sending tuples with the same key to the same worker node; combine: merging data from the same mapper function; partition: required, preparing the data from the mapper stage for distribution to the reducers.

[0125] The computing engine also includes a series of high-level interfaces, such as: project: reshape existing data, such as adding or removing fields; match: match filtering operations; group: grouping operations; search: full-text search; sort: sorting; union: merging; flatMap: flatten array-type fields; aggregate: general aggregation, which can be passed in user-defined functions.

[0126] For custom aggregate functions, binding to a programming language, such as Python, is supported, but the underlying computing engine needs to support the corresponding language's SDK.

[0127] The code executor submits the generated executable code to the target processing system for execution and waits for the target processing system to return the execution result. The code executor is implemented in a distributed manner because, generally speaking, individual task processing requests have no dependencies on each other, thus allowing for easy horizontal scaling.

[0128] The result collector is used to return the generated task execution results to the user. A task processing request may involve multiple stages from interpretation to execution, and the output of one stage may be the input of another stage. Depending on the final output of the task processing request, task processing requests can be further divided into the following categories based on their content:

[0129] Data definition: This involves the creation, deletion, updating, and querying of namespaces, datasets, processes, and tasks; that is, the data to be processed in a task processing request.

[0130] Data Manipulation: This involves the storage or computation logic of a dataset, i.e., the operations to be executed in a task processing request.

[0131] Object processing system: The object processing system accepts the executable code submitted by the code executor in the interpreter, executes the corresponding code logic in the engine, and returns the task execution result to the code executor. Object processing systems can be broadly categorized into storage systems and computing systems. Storage systems can be further divided into systems for Online Transaction Processing (OLTP) and systems for Online Analytical Processing (OLAP), while computing systems can be divided into offline batch processing systems and real-time computing systems.

[0132] Typical OLTP systems include: MySQL, Oracle, PostgreSQL, HBase, and Cassandra;

[0133] Typical OLAP systems include: Hive, Impala, ClickHouse, and Hadoop HDFS;

[0134] Typical offline batch processing systems include: Spark, Hadoop, and MapReduce;

[0135] Typical real-time computing systems include Storm and Flink.

[0136] Target processing systems are diverse and complex, each suited to different scenarios. By using the interface description language defined in this invention to write task processing requests and for the interpreter to translate these requests, the differences and details of these underlying systems can be shielded, providing users with a unified entry point. By registering with the interpreter and implementing the structure defined in the interface description language specification, target processing systems can easily utilize the comprehensive abstraction capabilities provided by the interface description language. This approach significantly lowers the barrier to entry for users and unlocks data productivity.

[0137] Based on the above description of the overall architecture of the data processing system, optionally, in step S102, determining the target processing system corresponding to the task processing request based on the task processing request may include: determining the target processing system corresponding to the task processing request through the code executor, based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

[0138] After receiving a task processing request through the interpreter, the code executor in the interpreter can determine the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

[0139] Figure 5 Flowchart of the data processing method provided in the embodiments of this application Figure 3 Optionally, in the above steps, determining the target processing system corresponding to the task processing request through the code executor, based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process, may include:

[0140] S501. Determine the data storage engine corresponding to the task processing request based on the source field of the dataset.

[0141] Optionally, the storage system corresponding to the task processing request can be determined based on the source field of the dataset, and the specific data storage engine corresponding to the task processing request can be determined based on the specific information of the source field.

[0142] For example, if the source field is "engine""mysql", it can be determined that the task processing request corresponds to the MySQL storage engine in the storage system.

[0143] S502. Based on the description field of the scheduler corresponding to the process and the operator field bound to the task, determine the data calculation engine corresponding to the task processing request.

[0144] Similarly, based on the description field of the scheduler corresponding to the process and the operator field bound to the task, it can be determined that the task request corresponds to a computing system, and a specific data computing engine in the corresponding computing system can be identified.

[0145] The scheduler is mainly responsible for scheduling tasks to specific physical computing nodes and allocating physical resources such as CPU and memory. Operators correspond to a specific data computing engine, such as Spark operators, Hive operators, and Flink operators. By combining the scheduler and operator information, the specific data computing engine to be called by the task processing request can be determined.

[0146] For example, if the description field of the scheduler corresponding to the process is "scheduler":"Airflow", it can be determined that the task processing request corresponds to the Airflow scheduler in the computing system. If the operator field bound to the task is "Spark operator", it can be determined that the task processing request corresponds to the Spark operator in the computing system. Therefore, it can be determined that the task processing request applies the Spark operator to execute the task in the physical computing node called by the Airflow scheduler.

[0147] S503, the target processing system includes a data storage engine and / or a data computing engine.

[0148] Optionally, the target processing system may include the data storage engine and data computing engine determined above. When the source field of the dataset is empty, the determined target processing system may only include the data computing engine. When the description field and operator field of the scheduler corresponding to the process are empty, the determined target processing system may only include the data storage engine.

[0149] Figure 6 Flowchart of the data processing method provided in the embodiments of this application Figure 4 Optionally, step S103, converting the task processing request into executable code corresponding to the target processing system and submitting it to the target processing system, may include:

[0150] S601. Convert the task processing request into the execution code corresponding to the target processing system through the predefined basic interface in the code generator.

[0151] Based on the pending operations in the task processing request, the corresponding basic interface can be determined, and then the task processing request can be converted into the execution code corresponding to the target processing system through each basic interface.

[0152] S602. Submit the executable code to the target processing system through the code executor.

[0153] The generated executable code can then be submitted to the target processing system via a code executor, where it will run and generate the task execution result corresponding to the task processing request.

[0154] In summary, the data processing method provided in this embodiment uses a predefined interface description language to write task processing requests. By parsing the key fields in the task processing request, the target processing system corresponding to the request can be determined. The task processing request is then converted into executable code supported by the target processing system. The target processing system runs the executable code to execute the task processing request and generates the task execution result. This method, by using a predefined interface description language to write task processing requests, allows different users to call various target processing systems within the data processing system simply by learning the unified interface description language. This significantly reduces the learning cost for users, lowers the learning threshold of the data processing system, and improves its versatility and usability.

[0155] The data capability system, implemented based on the defined interface description language, integrates data storage and computing capabilities, providing a unified perspective for operation, development, and use, achieving a good balance between performance and versatility. It supports more data models and provides clearer data access interfaces; it merges computing and storage systems, eliminating the need to design complex connectors for various storage systems. This data processing system simultaneously supports batch and streaming data processing and a rich set of computational operators, enabling users to easily expand and develop the data processing system based on existing operators.

[0156] The following describes the apparatus, equipment, and storage medium used to execute the data processing method provided in this application. The specific implementation process and technical effects are described above and will not be repeated here.

[0157] Figure 7 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. The functions implemented by this data processing apparatus correspond to the steps performed by the above-described method. This apparatus can be understood as a data processing system, which can be deployed on a computer device. Figure 7 As shown, the device may include: an acquisition module 710, a determination module 720, and an execution module 730;

[0158] Optionally, the acquisition module 710 is used to acquire the input task processing request, which includes: data to be processed and operations to be performed. The task processing request is written in a predefined interface description language.

[0159] The determination module 720 is used to determine the target processing system corresponding to the task processing request based on the task processing request.

[0160] The execution module 730 is used to convert the task processing request into the execution code corresponding to the target processing system, submit it to the target processing system, and run the execution code through the target processing system to generate the task execution result corresponding to the task processing request.

[0161] Optionally, the device may further include: a generation module;

[0162] The generation module is used to create language description key fields using a preset language format. The language description key fields include: namespace, dataset, process, and task.

[0163] Generate the interface description language based on the key fields described in each language.

[0164] Optionally, the namespace includes: a name field and a version field; the dataset includes: a name field and a source field; the process includes: a name field and a description field of the scheduler corresponding to the process; and the task includes: a name field and an operator field to which the task is bound.

[0165] Optionally, the acquisition module 710 is specifically used to generate a task processing request by inputting the information corresponding to each field in the language description key fields defined in the interface description language.

[0166] Optionally, the determining module 720 is specifically used to determine the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

[0167] Optionally, the data processing system includes: an interpreter and at least one object processing system, wherein the interpreter includes: a code generator and a code executor;

[0168] The determination module 720 is specifically used to determine the target processing system corresponding to the task processing request by means of the code executor, based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

[0169] Optionally, module 720 is specifically used to determine the data storage engine corresponding to the task processing request based on the source field of the dataset;

[0170] Based on the description field of the scheduler corresponding to the process and the operator field bound to the task, determine the data computing engine corresponding to the task processing request;

[0171] The target processing system includes a data storage engine and / or a data computing engine.

[0172] Optionally, the execution module 730 is specifically used to convert the task processing request into the execution code corresponding to the target processing system through the predefined basic interface in the code generator;

[0173] The code executor submits the executable code to the target processing system.

[0174] Through the aforementioned apparatus, after the acquisition module obtains a task processing request written in a predefined interface description language, the determination module, by parsing key fields in the task processing request, can determine the target processing system corresponding to the task processing request. The execution module then converts the task processing request into executable code supported by the target processing system. The target processing system runs the executable code to execute the task processing request and generate the task execution result. By using a predefined interface description language to write task processing requests, different users only need to learn the unified interface description language to call various target processing systems within the data processing system. This significantly reduces the learning cost for users, lowers the learning threshold of the data processing system, and improves its versatility and usability.

[0175] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0176] The modules described above can be connected or communicate with each other via wired or wireless connections. Wired connections can include metal cables, optical fibers, hybrid cables, or any combination thereof. Wireless connections can include connections via LAN, WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more modules can be combined into a single module, and any module can be divided into two or more units. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here.

[0177] Figure 8 A schematic diagram of an electronic device provided in this application embodiment includes: a processor 801, a storage medium 802, and a bus 803. The storage medium 802 stores machine-readable instructions executable by the processor 801. When the electronic device runs a data processing method as described in the embodiment, the processor 801 communicates with the storage medium 802 via the bus 803. The processor 801 executes the machine-readable instructions to perform the following steps:

[0178] The task processing request is obtained as input. The task processing request includes: the data to be processed and the operation to be performed. The task processing request is written in a predefined interface description language.

[0179] Based on the task processing request, determine the target processing system corresponding to the task processing request;

[0180] The task processing request is converted into the execution code corresponding to the target processing system, submitted to the target processing system, and the execution code is run by the target processing system to generate the task execution result corresponding to the task processing request.

[0181] In one feasible implementation, before executing the task processing request to obtain input, the processor 801 is specifically used to: create language description key fields using a preset format, the language description key fields including: namespace, dataset, process, and task;

[0182] Generate the interface description language based on the key fields described in each language.

[0183] In a feasible implementation, the namespace includes: a name field and a version field; the dataset includes: a name field and a source field; the process includes: a name field and a description field of the scheduler corresponding to the process; and the task includes: a name field and an operator field to which the task is bound.

[0184] In one feasible implementation, when the processor 801 executes the task processing request for obtaining input, it specifically performs the following: according to each field in the language description key fields defined in the interface description language, it inputs the information corresponding to each field and generates the task processing request.

[0185] In one feasible implementation, when the processor 801 performs the task processing request to determine the target processing system corresponding to the task processing request, it is specifically used to: determine the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

[0186] In one feasible implementation, when the processor 801 executes the task processing request to determine the target processing system corresponding to the task processing request, it specifically uses the code executor to determine the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

[0187] In one feasible implementation, when the processor 801 executes the code executor to determine the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process, it is specifically used to: determine the data storage engine corresponding to the task processing request based on the source field of the dataset.

[0188] Based on the description fields of the scheduler corresponding to the process, determine the data computing engine corresponding to the task processing request;

[0189] The target processing system includes a data storage engine and / or a data computing engine.

[0190] In one feasible implementation, when the processor 801 executes the conversion of the task processing request into executable code corresponding to the target processing system and submits it to the target processing system, it specifically converts the task processing request into executable code corresponding to the target processing system through a predefined basic interface in the code generator.

[0191] The code executor submits the executable code to the target processing system.

[0192] In this way, before the data processing system receives a task processing request written in a predefined interface description language, it can determine the target processing system corresponding to the task processing request by parsing the key fields in the request. This allows the task processing request to be converted into executable code supported by the target processing system. The target processing system then runs the executable code to execute the task processing request and generate the task execution result. This method, by using a predefined interface description language to write task processing requests, allows different users to call various target processing systems within the data processing system simply by learning the unified interface description language. This significantly reduces the learning cost for users, lowers the learning threshold of the data processing system, and improves its versatility and performance.

[0193] The storage medium 802 stores program code, which, when executed by the processor 801, causes the processor 801 to perform various steps in the data processing methods according to various exemplary embodiments of this application as described in the "Exemplary Methods" section above.

[0194] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0195] Storage medium 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The storage medium 802 in this embodiment can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0196] Optionally, embodiments of this application also provide a computer-readable storage medium storing a computer program, which is executed by a processor, and the processor performs the following steps:

[0197] The task processing request is obtained as input. The task processing request includes: the data to be processed and the operation to be performed. The task processing request is written in a predefined interface description language.

[0198] Based on the task processing request, determine the target processing system corresponding to the task processing request;

[0199] The task processing request is converted into the execution code corresponding to the target processing system, submitted to the target processing system, and the execution code is run by the target processing system to generate the task execution result corresponding to the task processing request.

[0200] In one feasible implementation, before executing the task processing request to obtain input, the processor 801 is specifically used to: create language description key fields using a preset format, the language description key fields including: namespace, dataset, process, and task;

[0201] Generate the interface description language based on the key fields described in each language.

[0202] In a feasible implementation, the namespace includes: a name field and a version field; the dataset includes: a name field and a source field; the process includes: a name field and a description field of the scheduler corresponding to the process; and the task includes: a name field and an operator field to which the task is bound.

[0203] In one feasible implementation, when the processor 801 executes the task processing request for obtaining input, it specifically performs the following: according to each field in the language description key fields defined in the interface description language, it inputs the information corresponding to each field and generates the task processing request.

[0204] In one feasible implementation, when the processor 801 performs the task processing request to determine the target processing system corresponding to the task processing request, it is specifically used to: determine the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

[0205] In one feasible implementation, when the processor 801 executes the task processing request to determine the target processing system corresponding to the task processing request, it specifically uses the code executor to determine the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

[0206] In one feasible implementation, when the processor 801 executes the code executor to determine the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process, it is specifically used to: determine the data storage engine corresponding to the task processing request based on the source field of the dataset.

[0207] Based on the description fields of the scheduler corresponding to the process, determine the data computing engine corresponding to the task processing request;

[0208] The target processing system includes a data storage engine and / or a data computing engine.

[0209] In one feasible implementation, when the processor 801 executes the conversion of the task processing request into executable code corresponding to the target processing system and submits it to the target processing system, it specifically converts the task processing request into executable code corresponding to the target processing system through a predefined basic interface in the code generator.

[0210] The code executor submits the executable code to the target processing system.

[0211] In this way, before the data processing system receives a task processing request written in a predefined interface description language, it can determine the target processing system corresponding to the task processing request by parsing the key fields in the request. This allows the task processing request to be converted into executable code supported by the target processing system. The target processing system then runs the executable code to execute the task processing request and generate the task execution result. This method, by using a predefined interface description language to write task processing requests, allows different users to call various target processing systems within the data processing system simply by learning the unified interface description language. This significantly reduces the learning cost for users, lowers the learning threshold of the data processing system, and improves its versatility and performance.

[0212] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0216] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data processing method, characterized in that, Applied to a data processing system, the method includes: The system acquires an input task processing request, which includes: data to be processed and operations to be performed. The task processing request is written using a predefined interface description language. This interface description language is generated based on created language description key fields, which define data storage abstractions and data computation abstractions. These language description key fields include: namespace, dataset, process, and task. The namespace includes: a name field and a version field. The dataset includes: a name field and a source field. The process includes: a name field and a description field of the corresponding scheduler. The task includes: a name field and an operator field to which the task is bound. Based on the task processing request, the target processing system corresponding to the task processing request is determined; the target processing system includes a data storage engine and a data computing engine; The task processing request is converted into executable code corresponding to the target processing system and submitted to the target processing system. The executable code is then run by the target processing system to generate the task execution result corresponding to the task processing request. The step of determining the target processing system corresponding to the task processing request based on the task processing request includes: Based on the source field of the dataset, determine the data storage engine corresponding to the task processing request; Based on the description field of the scheduler corresponding to the process and the operator field bound to the task, determine the data computing engine corresponding to the task processing request; The target processing system includes the data storage engine and / or the data computing engine.

2. The method according to claim 1, characterized in that, Before the task processing request for obtaining input, the following are included: Create language description key fields using a preset language format; The interface description language is generated based on the key fields described in each language.

3. The method according to claim 1 or 2, characterized in that, The task processing request for obtaining input includes: According to the key fields in the interface description language, input the information corresponding to each field to generate the task processing request.

4. The method according to claim 1 or 2, characterized in that, The step of determining the target processing system corresponding to the task processing request based on the task processing request includes: The target processing system corresponding to the task processing request is determined based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

5. The method according to claim 4, characterized in that, The data processing system includes: an interpreter and at least one target processing system, wherein the interpreter includes: a code generator and a code executor; determining the target processing system corresponding to the task processing request based on the task processing request includes: The code executor determines the target processing system corresponding to the task processing request based on the source field of the dataset in the task processing request and / or the description field of the scheduler corresponding to the process.

6. The method according to claim 5, characterized in that, The step of converting the task processing request into executable code corresponding to the target processing system and submitting it to the target processing system includes: The task processing request is converted into the execution code corresponding to the target processing system through the predefined basic interface in the code generator; The code executor submits the executable code to the target processing system.

7. A data processing apparatus, characterized in that, The device, used in a data processing system, includes: an acquisition module, a determination module, and an execution module. The acquisition module is used to acquire input task processing requests. The task processing request includes: data to be processed and operations to be performed. The task processing request is written using a predefined interface description language. The interface description language is generated based on created language description key fields. These language description key fields define data storage abstractions and data computation abstractions. The language description key fields include: namespace, dataset, process, and task. The namespace includes: a name field and a version field. The dataset includes: a name field and a source field. The process includes: a name field and a description field of the corresponding scheduler. The task includes: a name field and an operator field to which the task is bound. The determining module is used to determine the target processing system corresponding to the task processing request based on the task processing request; the target processing system includes a data storage engine and a data computing engine; The execution module is used to convert the task processing request into execution code corresponding to the target processing system, submit it to the target processing system, and run the execution code through the target processing system to generate the task execution result corresponding to the task processing request; The determining module is specifically used to determine the data storage engine corresponding to the task processing request based on the source field of the dataset; and to determine the data computing engine corresponding to the task processing request based on the description field of the scheduler corresponding to the process and the operator field bound to the task. The target processing system includes the data storage engine and / or the data computing engine.

8. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is executed by a processor to perform the data processing method as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Big data processing method and device

    CN114925035A