Data processing methods, apparatus, electronic devices and storage media

Abstract

This disclosure relates to a data processing method, apparatus, electronic device, and storage medium. The method includes: when a logical table corresponding to a data source is detected to be nonexistent, obtaining metadata of the data source; obtaining second schema information based on the metadata and data table structure adjustment information; obtaining SQL statement information corresponding to the logical table registered by the second schema information; creating a target data table matching the current task environment based on the target schema information corresponding to the logical table name obtained by parsing the SQL statement information; and processing the target data table according to the operation information obtained by parsing the SQL statement information. This greatly facilitates users in developing real-time tasks based on real-time computing platforms. Based on this disclosure, the difficulty of real-time computation on data sources without schemas can be reduced, and the feasibility of real-time computation can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of big data, real-time computing is a crucial technology, widely used in real-time processing, real-time reporting, real-time monitoring and alarms, real-time algorithm model training, and real-time recommendations, thereby improving the effectiveness of real-time data acquisition, analysis, and decision-making. SQL (Structured Query Language) is the most widely used development language in the big data field, significantly lowering the barrier to entry for real-time development.

[0003] However, data sources for real-time computation based on SQL, such as the distributed publish-subscribe messaging system Kafka, or other databases that require access to remote data services like Redis (Remote Dictionary Server) and distributed storage systems like HBase (Hadoop Database), are data sources without a schema (data table structure), making it impossible to directly write SQL for real-time computation.

[0004] In related technologies, to achieve real-time computation on schema-less data sources, users typically need to write SQL DDL (Data Definition Language) to define the table schema. This approach has at least two drawbacks: 1) Writing DDL is cumbersome and error-prone, especially when there are many fields. 2) DDL support is not very good if the data source schema is multi-level nested or consists of complex types such as structs or arrays. These shortcomings increase the difficulty and reduce the feasibility of real-time computation on schema-less data sources. Summary of the Invention

[0005] This disclosure provides a data processing method, apparatus, electronic device, and storage medium to at least solve the problems of cumbersome writing process, error-proneness, and lack of support for multi-level nesting or complex data types caused by defining table schemas through user-written SQL DDL in related technologies. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a data processing method is provided, comprising:

[0007] When a logical table corresponding to a data source is detected as not existing, the metadata of the data source is obtained. The metadata includes basic data information and a data definition file. The data definition file is used to define the serialization and deserialization mechanism of the data source.

[0008] The basic information of the data is analyzed according to the data definition file to obtain the first data table structure schema information corresponding to the data source;

[0009] Obtain data table structure adjustment information for the first schema information, and based on the first schema information and the data table structure adjustment information, obtain second schema information; the second schema information includes field names and field types;

[0010] Based on the field names and field types in the second schema information, the data source is registered as a logical table;

[0011] Obtain the Structured Query Language (SQL) statement information corresponding to the logical table, parse the SQL statement information to obtain the logical table name and the information to be operated, and obtain the target schema information corresponding to the logical table name;

[0012] Based on the target schema information, the data source is created as a target data table that matches the current task environment;

[0013] The target data table is processed according to the information to be operated.

[0014] As an optional implementation, obtaining the metadata of the data source includes:

[0015] If no registered data source matches the data source, obtain the data source information corresponding to the data source.

[0016] Register the data source according to the data source information, and obtain the basic data information corresponding to the registered data source;

[0017] Obtain the data definition file corresponding to the data source, and determine the metadata of the data source based on the data definition file and the basic data information.

[0018] As an optional implementation, obtaining the metadata of the data source includes:

[0019] If a registered data source that matches the data source exists, bind the registered data source to the data source;

[0020] Obtain basic data information from the registered data source;

[0021] Obtain the data definition file corresponding to the data source, and determine the metadata corresponding to the registered data source based on the data definition file and the basic data information. The metadata corresponding to the registered data source is the metadata of the data source.

[0022] As an optional implementation, after the step of registering the data source as a logical table based on the field names and field types in the second schema information, the method further includes:

[0023] Register the second schema information and corresponding logical table name of the data source to the schema service;

[0024] The second schema information in the schema service is updated based on the real-time schema information corresponding to the data source stored in the metadata center.

[0025] As an optional implementation, the step of obtaining the Structured Query Language (SQL) statement information corresponding to the logical table includes:

[0026] Display the second schema information corresponding to the logical table;

[0027] Obtain information about structured language SQL statements written based on the displayed second schema information.

[0028] As an optional implementation, the step of processing the target data table based on the information to be operated includes:

[0029] Dynamically parse the data from the data source based on the field names and field types corresponding to the target data table;

[0030] The processing instructions indicated by the pending operation information are executed on the parsed data.

[0031] As an optional implementation, before obtaining the metadata of the data source when a logical table corresponding to the data source is detected to be nonexistent, the method further includes:

[0032] If a logical table corresponding to a data source is detected, the target schema information corresponding to the logical table is obtained from the schema service.

[0033] According to a second aspect of the present disclosure, a data processing apparatus is provided, comprising:

[0034] The first acquisition module is configured to acquire the metadata of the data source when it is detected that no logical table corresponding to the data source exists. The metadata includes basic data information and a data definition file. The data definition file is used to define the serialization and deserialization mechanism of the data source.

[0035] The second acquisition module is configured to perform analysis of the basic information of the data according to the data definition file, and acquire the first data table structure schema information corresponding to the data source;

[0036] The third acquisition module is configured to acquire data table structure adjustment information for the first schema information, and obtain second schema information based on the first schema information and the data table structure adjustment information; the second schema information includes field names and field types;

[0037] The registration module is configured to register the data source as a logical table based on the field names and field types in the second schema information.

[0038] The parsing module is configured to execute and obtain the Structured Query Language (SQL) statement information corresponding to the logical table, and parse the SQL statement information to obtain the logical table name and the information to be operated, and obtain the target schema information corresponding to the logical table name;

[0039] The first processing module is configured to create a target data table that matches the current task environment based on the target schema information;

[0040] The second processing module is configured to process the target data table according to the information to be operated.

[0041] As an optional implementation, the first acquisition module includes:

[0042] The first acquisition submodule is configured to retrieve the data source information corresponding to the data source if no registered data source matching the data source exists.

[0043] The second acquisition submodule is configured to register the data source based on the data source information and acquire the basic data information corresponding to the registered data source.

[0044] The first determining submodule is configured to retrieve the data definition file corresponding to the data source, and determine the metadata of the data source based on the data definition file and the basic data information.

[0045] As an optional implementation, the first acquisition module includes:

[0046] The binding submodule is configured to bind the registered data source to the data source if a registered data source matching the data source exists.

[0047] The third acquisition submodule is configured to acquire basic data information of the registered data source.

[0048] The second determining submodule is configured to retrieve the data definition file corresponding to the data source, and determine the metadata corresponding to the registered data source based on the data definition file and the basic data information, wherein the metadata corresponding to the registered data source is the metadata of the data source.

[0049] As an optional implementation, the device further includes:

[0050] The registration module is configured to register the second schema information and the corresponding logical table name of the data source to the schema service.

[0051] The update module is configured to update the second schema information in the schema service based on the real-time schema information corresponding to the data source stored in the metadata center.

[0052] As an optional implementation, the parsing module includes:

[0053] The display submodule is configured to display the second schema information corresponding to the logical table;

[0054] The fourth acquisition submodule is configured to retrieve structured language SQL statements based on the displayed second schema information.

[0055] As an optional implementation, the parsing module includes:

[0056] The fifth acquisition submodule is configured to retrieve the target schema information corresponding to the logical table name from the schema service.

[0057] As an optional implementation, the second processing module includes:

[0058] The first processing submodule is configured to dynamically parse the data from the data source based on the field names and field types corresponding to the target data table;

[0059] The second processing submodule is configured to execute the processing instructions indicated by the operation information on the parsed data.

[0060] As an optional implementation, the device further includes:

[0061] The fourth acquisition module is configured to retrieve the target schema information corresponding to the logical table from the schema service when a logical table corresponding to the data source is detected.

[0062] According to a third aspect of the present disclosure, a storage medium is provided that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform a data processing method as described in any of the above embodiments.

[0063] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0064] processor;

[0065] Memory used to store the processor's executable instructions;

[0066] The processor is configured to execute the instructions to implement the data processing method as described in any of the above embodiments.

[0067] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the data processing method provided in any of the above embodiments.

[0068] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0069] This embodiment of the disclosure, when detecting the absence of a logical table corresponding to a data source, obtains the data source's metadata; based on the metadata and data table structure adjustment information, obtains the second schema information; obtains the SQL statement information corresponding to the logical table registered with the second schema information; based on the target schema information corresponding to the logical table name obtained by parsing the SQL statement information, creates the data source as a target data table matching the current task environment; and processes the target data table according to the operation information obtained by parsing the SQL statement information. This greatly facilitates users in developing real-time tasks on real-time computing platforms similar to Flink SQL. The entire process of data source registration and table usage is simple, with automatic loading and updating avoiding manual operations; furthermore, the registered data source can be shared and used, and it can function as a logical table, facilitating data governance, such as lineage relationships and data metadata, reducing the difficulty of real-time computation on data sources without a schema, and improving the efficiency and feasibility of real-time computation.

[0070] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0071] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0072] Figure 1 This is a schematic diagram illustrating the implementation environment of a data processing method according to an exemplary embodiment.

[0073] Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment.

[0074] Figure 3 This is a flowchart illustrating a data processing method according to an exemplary embodiment.

[0075] Figure 4 This is a flowchart illustrating a data processing method according to an exemplary embodiment.

[0076] Figure 5 This is a partial flowchart illustrating a data processing method according to an exemplary embodiment.

[0077] Figure 6 This is a schematic flowchart of a data processing apparatus according to an exemplary embodiment.

[0078] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0079] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0080] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0081] Before providing a further detailed description of the embodiments of this disclosure, the nouns and terms involved in the embodiments of this disclosure will be explained, and the nouns and terms involved in the embodiments of this disclosure shall be interpreted as follows.

[0082] SQL (Structured Query Language) is a database query and programming language used to access, query, update, and manage relational database systems.

[0083] Schema: This is the data table structure, which is a collection of database objects. Schema contains various objects such as tables, indexes, and views.

[0084] No schema: This means no table structure is used, that is, no data table structure is used, and the data structure has a high degree of freedom.

[0085] DDL (Data Definition Language) refers to SQL statements used when creating tables, such as CREATE, ALTER, and DROP. DDL is primarily used for defining or changing table structure, data types, table links, and constraints during initialization.

[0086] Data source: The source of data used to generate streaming data or static datasets. For example, a data source may include any one of the following: a distributed publish-subscribe messaging system Kafka, a remote data service Redis, a distributed storage system HBase, etc. Of course, the data sources involved in this disclosure are not limited to these, and may also include any one of the following: the Hadoop Distributed File System HDFS, Flume, etc.

[0087] Flink refers to a system that supports stream processing and further enhances real-time analysis. As a distributed processing engine for streaming and batch data, Flink possesses the basic characteristics of high throughput, low latency, and high performance in data processing, and also supports various relatively flexible window semantics such as time, session, and counting.

[0088] Flink SQL is a development language designed by Flink Real-Time Computing to simplify the computing model and lower the barrier to entry for users to use real-time computing, conforming to standard SQL semantics.

[0089] Figure 1 This is a schematic diagram illustrating the implementation environment of a data processing method according to an exemplary embodiment. See also: Figure 1 The implementation environment may include a data processing system 10 and a terminal 20.

[0090] The data processing system 10 can be used to provide background data processing services, such as real-time computing services and real-time statistical services. The terminal 20 can be used to obtain the SQL statement input by the user and send the SQL statement to the data processing system 10 to obtain the data processing service results from the data processing system 10.

[0091] The data processing system 10 described above can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 20 described above can also be a physical device such as a smartphone, tablet, laptop, desktop computer, smart speaker, smart wearable device, digital assistant, augmented reality device, virtual reality device, or an application running on a physical device, but is not limited thereto. The data processing system 10 and the terminal 20 can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any limitations.

[0092] For example, continue as follows Figure 1 As shown, the data processing system 10 may include a real-time computing platform 11, a metadata center 12, and a schema service 13. Among them,

[0093] The real-time computing platform 11 can be used at least as a platform capable of real-time computation and processing of streaming data, such as the Flink SQL real-time computing platform. Here, the Flink SQL real-time computing platform allows users to write SQL statements for real-time Flink computation, utilizing various data sources without table structures, such as tables abstracted from Kafka, Redis, and HBase.

[0094] Metadata center 12 can be used to store metadata information for various data sources. Taking Kafka as an example, the metadata information can include at least one of the following: the cluster name where Kafka resides, the topic name, the schema type (e.g., Protocol Buffer, JSON), and the data definition file (e.g., Protocol Buffer definition file). Users can search for the corresponding data source based on the metadata information.

[0095] Schema service 13 can be used at least for registering and managing the schema corresponding to the tables of the data source abstraction.

[0096] It should be noted that the structural framework of the data processing system 10 described above is not limited to this, and may also be other structural frameworks. For example, the functions of the metadata center 12 can also be integrated into the real-time computing platform 11, which only needs to store these metadata information, and / or the functions of the schema service 13 can also be integrated into the real-time computing platform 11. This disclosure does not impose any restrictions here.

[0097] Typically, data sources for real-time computation based on SQL, such as the distributed publish-subscribe messaging system Kafka, or other databases requiring access to remote data services like Redis (Remote Dictionary Server) and distributed storage systems like HBase (Hadoop Database), are schema-less data sources, making it impossible to directly write SQL for computation. In related technologies, to achieve real-time computation on schema-less data sources, users need to write SQL DDL (Data Definition Language) to define the table schema. However, implementing SQL computation on schema-less data sources through DDL is not only cumbersome and error-prone, especially with many fields, requiring a large number of SQL statements and increasing the likelihood of errors, but DDL also does not support multi-level nested types (e.g., nested tables, nested arrays) or other complex data types (e.g., object types, array types, struct types), which to some extent affects the difficulty and feasibility of real-time computation on schema-less data sources, thus limiting its application. Therefore, this disclosure provides a data processing method, apparatus, electronic device, and storage medium to at least solve one of the above-mentioned technical problems.

[0098] Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment, such as... Figure 2 As shown, the data processing method can be applied to electronic devices or data processing apparatuses and includes the following steps.

[0099] In step S201, when it is detected that there is no logical table corresponding to the data source, the metadata of the data source is obtained. The metadata includes basic data information and data definition file.

[0100] The data definition file is used to define the serialization and deserialization mechanisms of the data source.

[0101] In one implementation, the data source includes the target data source that needs to be used without a data table structure, such as, but not limited to, the distributed publish-subscribe messaging system Kafka, or other systems that require access to remote data services such as Redis (Remote Dictionary Server) and distributed storage systems such as HBase (Hadoop Database). Fields in these data sources typically store data values ​​serialized as byte arrays like 0101. Because they lack field names and field types, tables cannot be formed, and therefore, SQL cannot be written for calculations.

[0102] Metadata is used to reflect metadata information of the data source. For example, the metadata here includes at least basic data information and data definition files.

[0103] The basic information of this data includes, but is not limited to, the cluster name where the data source resides, the topic name, and the schema type of the data source. Taking Kafka as an example, the cluster name where the data source resides can be the same as the Kafka cluster name. The schema type of the data source can include, but is not limited to, Protocol Buffer, JSON, etc.

[0104] This data definition file defines the serialization and deserialization mechanisms of the data source. Serialization refers to the process of converting structured data into a binary data sequence, such as a byte array (like 0s and 1s), while deserialization refers to the process of converting the serialized binary data sequence into a data structure that is easy to process and read / write. For example, this data definition file can include, but is not limited to, Protocol Buffer definition files, JSON definition files, and Thrift definition files. The data definition file is determined based on the schema type of the data source in its metadata. Through this data definition file, the field names and field types of each byte array in a data source without a table structure can be determined, and thus the specific values ​​corresponding to each field name and field type can be determined.

[0105] The logical table here is a table with defined data, including field names and field types. In one embodiment, the logical table is used to display data to a user, allowing the user to write SQL statements based on the field names and field types of the logical table. In another embodiment, the logical table is used by an electronic device to generate SQL statements based on the field names and field types of the logical table.

[0106] In one embodiment, step S201, which involves obtaining the metadata of the data source, can be divided into the following two cases.

[0107] In the first scenario: In an optional approach, obtaining the metadata of the data source may include:

[0108] In step S301, if there is no registered data source matching the data source, the data source information corresponding to the data source is obtained;

[0109] In step S302, the data source is registered according to the data source information, and the basic data information corresponding to the registered data source is obtained;

[0110] In step S303, the data definition file corresponding to the data source is obtained, and the metadata of the data source is determined based on the data definition file and the basic data information.

[0111] As an example only, when the real-time computing platform determines that there is no logical table corresponding to the data source, it checks whether there is a registered data source matching the data source in the metadata center. If no registered data source matching the data source exists, it obtains the data source information corresponding to the data source. This data source information includes, but is not limited to, basic information about the data source and links to files or code corresponding to the schema, so as to obtain and register the data source based on this information. Next, it obtains the basic data information corresponding to the registered data source and the data definition file corresponding to the data source, and determines the metadata of the data source based on the data definition file and the basic data information. The data definition file can correspond to the data source and can be predefined and stored according to the data source.

[0112] In the second scenario: In an optional approach, obtaining the metadata of the data source includes:

[0113] In step S301', if there is a target registered data source that matches the data source, the registered data source is bound to the data source;

[0114] In step S302', the basic information of the registered data source is obtained;

[0115] In step S303', the data definition file corresponding to the data source is obtained, and the metadata corresponding to the registered data source is determined according to the data definition file and the basic data information. The metadata corresponding to the registered data source is the metadata of the data source.

[0116] As an example only, when the real-time computing platform determines that there is no logical table corresponding to the data source, it checks whether there is a registered data source matching the data source in the metadata center. If a registered data source matching the data source exists, the platform binds the registered data source to the data source, then obtains the basic data information corresponding to the registered data source and the data definition file corresponding to the data source, and determines the metadata corresponding to the registered data source based on the data definition file and the basic data information. Finally, the metadata corresponding to the registered data source is used as the metadata of the data source. The data definition file can also correspond to the data source, and it can be predefined and stored according to the data source.

[0117] In the above embodiments, when it is determined that no logical table exists corresponding to the data source, binding is performed when a matching registered data source is found. Then, the metadata of the data source is determined based on this registered data source, reducing duplicate registration of data sources, further reducing the amount of data processing computation, and improving real-time processing efficiency. When it is determined that no matching registered data source exists, the data source is automatically registered based on the corresponding data source information, improving the scalability of data processing and reducing manual operations.

[0118] In step S203, the basic information of the data is analyzed according to the data definition file to obtain the first data table structure schema information corresponding to the data source.

[0119] In one embodiment, based on the data definition file in the metadata, since the serialization and deserialization mechanism of the data source is defined in the data definition file, the basic information of the data can be analyzed based on the data definition file, the byte array of each field in the data source can be converted into the corresponding data value, and then the field name and field type can be obtained. Based on the field name and field type, the first data table structure schema information corresponding to the data source can be generated.

[0120] In step S205, data table structure adjustment information for the first schema information is obtained, and second schema information is obtained based on the first schema information and the data table structure adjustment information; the second schema information includes field names and field types.

[0121] In one implementation, after obtaining the first schema information corresponding to the data source, the first schema information can be displayed to the user so that the user can confirm whether the first schema information matches. Specifically, the user can be shown content including but not limited to field names, field types, and field descriptions to facilitate the user's quick confirmation of whether the obtained first schema information matches. If a confirmation operation confirming that the first schema information matches is received, the data structure of the displayed first schema information can be adjusted, such as, but not limited to, deleting fields, modifying field names, modifying field descriptions, or setting fields as primary keys.

[0122] This application embodiment implements the adjustment operation of the first schema information, that is, the data structure of the first schema information is adjusted, so as to obtain the second schema information based on the first schema information and the data table structure adjustment information. The second schema information includes field names and field types.

[0123] Of course, in some cases, no data structure adjustment operation needs to be performed on the first schema information, and in this case, the first schema information and the second schema information can be the same.

[0124] In step S207, the data source is registered as a logical table based on the field names and field types in the second schema information.

[0125] As an example only, after performing data structure adjustments on the first schema information, a registration button can be triggered, causing the real-time computing platform to register the data source as a logical table. When the real-time computing platform receives the registry instruction, it can obtain the first schema information and data structure adjustment information corresponding to the data source, i.e., obtain the second schema information, and register the data source as a logical table based on the field names and field types in the second schema information. This logical table can be understood as a view in a database; it is a virtual table that can be mapped to a single physical table or composed of multiple physical tables from different data sources. The registered logical table can consist of data sources corresponding to fields with the same meaning.

[0126] In an optional implementation, after step S207, which involves registering the data source as a logical table based on the field names and field types in the second schema information, the method may further include:

[0127] In step S401, the second schema information of the data source and the corresponding logical table name are registered to the schema service;

[0128] In step S403, the second schema information in the schema service is updated based on the real-time schema information corresponding to the data source stored in the metadata center.

[0129] As an example only, when registering a data source as a logical table, the second schema information of the data source and its corresponding logical table name can be bound and registered in the schema service. For example, the registration content in the schema service can be represented as: {(Schema_1,name_1),(Schema_2,name_2),.......,(Schema_n,name_n)}, where each piece of second schema information (Schema_1~Schema_n) is bound to its corresponding logical table name (name_1~name_n). Since the second schema information is bound to its corresponding logical table name, a one-to-one correspondence is established between the two, so that the corresponding second schema information can be determined based on the table name of the logical table later. Here, the schema service can be an application programming interface (API) used to store schema information.

[0130] In this embodiment, the schema information in the schema service is dynamically updated periodically or as needed, according to actual requirements. During the dynamic update process, the schema service can obtain real-time schema information of the data source from the metadata center, and dynamically update the second schema information in the schema service based on the real-time schema information, that is, update the second schema information in the schema service to the obtained real-time schema information.

[0131] In the above embodiments, the storage and updating of basic data source information are performed in the metadata center. The real-time computing platform obtains the metadata of the data source and dynamically parses the real-time schema information of the metadata, thus avoiding the need for users to manually change the field names and field types of the data source. In practical applications, the real-time computing platform can obtain the schema information of the data source from the schema service. To ensure the consistency between the actual schema of the data source (stored in the metadata center) and the schema information used by the user on the real-time computing platform, the schema service can periodically or automatically obtain the real-time schema information corresponding to the data source in the metadata center and update it, so that users can obtain the latest data source in real time. Furthermore, since the storage and updating of basic data source information are performed in the metadata center, the schema information obtained by the user based on the real-time computing platform is consistent with the actual schema information in the metadata center, facilitating data sharing among multiple users for tables registered with the schema service.

[0132] In step S209, the Structured Query Language (SQL) statement information corresponding to the logical table is obtained, and the SQL statement information is parsed to obtain the logical table name and the information to be operated on, thereby obtaining the target schema information corresponding to the logical table name. It can be understood that the target schema information is the latest schema information corresponding to the data source.

[0133] In one embodiment, obtaining the Structured Query Language (SQL) statement information corresponding to the logical table may include:

[0134] Display the second schema information corresponding to the data source, which includes at least field names and field types;

[0135] Obtain information about structured language SQL statements written based on the displayed second schema information.

[0136] As an example only, the real-time computing platform can display the second schema information corresponding to the data source to the user, or it can display a logical table containing the second schema information. This second schema information includes at least field names and field types, and may also include the logical table name, data source name, and data source type. Specifically, the second schema information corresponding to the data source can be displayed to the user on a web interface, allowing the user to write structured language SQL statements based on this second schema information, which the real-time computing platform can then retrieve.

[0137] In another real-time mode, obtaining the Structured Query Language (SQL) statement information corresponding to the logical table may include: obtaining the second schema information corresponding to the data source, wherein the second schema information includes at least field names and field types; and generating the structured language SQL statement information based on the second schema information.

[0138] As an example only, the real-time computing platform can automatically obtain the second schema information corresponding to the data source. This second schema information includes at least field names and field types, and may also include logical table names, data source names, and data source types. Based on the field names and field types in the obtained second schema information, the real-time computing platform automatically generates at least one syntactically correct initial SQL statement. This initial SQL statement can then be displayed to the user, who can confirm the final SQL statement by selecting or confirming an action. This allows the real-time computing platform to execute subsequent operations based on the final SQL statement. In this way, automatically generating SQL statements reduces user steps and improves data processing efficiency.

[0139] In one implementation, the Structured Query Language (SQL) statement information corresponding to the logical table is obtained, and the SQL statement information is parsed to obtain the logical table name and the information to be operated on. Next, the target schema information corresponding to the logical table name can be obtained from the schema service; this target schema information is the current schema information corresponding to the data source. Thus, by registering the second schema information of the data source and the corresponding logical table name to the schema service, and dynamically updating the schema information in the schema service, a connection is established between the schema service and the real-time computing platform. This allows for convenient and rapid retrieval of the corresponding target schema information from the schema service based on the logical table name parsed by the real-time computing platform, improving real-time computing efficiency.

[0140] As an example, if the obtained SQL statement information is sql="select ab from table_1", then the corresponding logical table name table_1 is obtained by parsing this SQL statement information. Then, table_1 is used to retrieve the schema information corresponding to the logical table name from the schema service. Since the schema information stored in the schema service and the data source corresponding to the metadata center are dynamically updated, meaning the schema information in the schema service may have been adjusted, the schema information obtained from the schema service here can be the second schema information or the updated second schema information. Next, the database type corresponding to the data source (e.g., at least one type such as Kafka, Redis, and HBase) is obtained. Then, the target schema information corresponding to the data source is determined based on the merged schema information and database type. This target schema information includes a richer data structure and better reflects the actual data information corresponding to the data source.

[0141] The above embodiments determine the target schema information by obtaining the database type corresponding to the data source and combining it with the schema information corresponding to the logical table name obtained by parsing the SQL statement. This makes the target schema information include a richer data structure, better reflect the real data information corresponding to the data source, and facilitate the creation of the data source as a target data table that matches the current task environment.

[0142] In step S211, based on the target schema information, the data source is created as a target data table that matches the current task environment.

[0143] The current task environment refers to the environment in which a real-time computing platform runs, which may include, but is not limited to, Flink SQL, Storm, and Spark platforms. The target data table includes tables that the real-time computing platform environment depends on.

[0144] For example, taking the Flink SQL platform as the real-time computing platform, the target data table is a table that the Flink SQL platform can recognize. This table can be represented as a Flink table, and the current task environment is a Flink table environment (TableEnvironment). Specifically, the Flink SQL platform obtains the target schema information corresponding to the data source, determines the database type corresponding to the target schema information, and then obtains the actual data corresponding to the data source based on this database type. Furthermore, based on the data format (e.g., JSON, Csv, Protocol Buffer, etc.), field names, and field types provided by the target schema information, and the actual data, it uses methods such as createTemporary() in the TableEnvironment within the current TableEnvironment environment to create a target data table that matches the specified output end and the current task environment—that is, a Flink table that the Flink SQL platform can recognize. Because this Flink table is a real table in the Flink environment, it can be used for real-time computation.

[0145] In step S213, the target data table is processed according to the information to be operated.

[0146] In one embodiment, the target data table includes tables on which the real-time computing platform environment depends. The step of processing the target data table according to the operation information may include: dynamically parsing data from the data source based on the field names and field types corresponding to the target data table; and executing the processing instructions indicated by the operation information on the parsed data.

[0147] Taking the Flink SQL platform as an example, the Flink SQL platform dynamically parses data from the data source based on the Flink table; then, based on the information to be operated, it performs real-time processing on the parsed data. This real-time processing includes, but is not limited to, at least one of real-time querying, real-time computation, real-time statistics, real-time search, real-time monitoring, and real-time recommendation.

[0148] This embodiment of the disclosure, when detecting the absence of a logical table corresponding to a data source, obtains the metadata of the data source; obtains the second schema information based on the metadata and data table structure adjustment information; obtains the SQL statement information corresponding to the logical table registered by the second schema information; based on the target schema information corresponding to the logical table name obtained by parsing the SQL statement information, creates the data source as a target data table matching the current task environment; and processes the target data table according to the operation information obtained by parsing the SQL statement information. This greatly facilitates users in developing real-time tasks on real-time computing platforms similar to Flink SQL. The entire process of registering and using tables for the data source is simple, with automatic loading and updating avoiding manual operations; furthermore, the data source can be shared after registration, and the registered data source can be a logical table, facilitating data governance, such as lineage relationships (e.g., same-source same-database lineage, same-source different-database lineage, different-source data lineage, etc.), data metadata governance, etc., reducing the difficulty of real-time computation on data sources without a schema, and improving the efficiency and feasibility of real-time computation, while also supporting data processing of data types including multi-level nested types.

[0149] Furthermore, information from the metadata center is acquired and dynamically parsed. Basic information from various data sources is stored and updated in a single location: the metadata center. The real-time computing platform acquires the metadata from the data sources and dynamically parses the schema information. This avoids users manually filling in or changing field names and types. The schema service needs to periodically or automatically acquire and update the real-time schema information corresponding to the data sources in the metadata center. This ensures users have access to the latest data sources in real time, guaranteeing consistency between the actual schema of the data sources and the schema information used by users on the real-time computing platform. Simultaneously, since the storage and updating of basic data source information are performed in the metadata center, the schema information obtained by users from the real-time computing platform is synchronized with the actual schema information in the metadata center, facilitating data sharing among multiple users for tables registered with the schema service. The metadata information of the data sources is stored in the metadata center, facilitating the statistics and management of various data sources, such as the lineage relationships between data sources and the relationships between data sources and logical tables, greatly simplifying data governance. Data sources can be easily explored and previewed. The schema information of the data sources is managed in the metadata center and the schema service, allowing users to easily preview the content of data sources, such as Kafka messages, on the data development platform. Normally, previewing Kafka messages requires writing separate programs. However, this application eliminates the need for manual programming. It allows users to preview Kafka messages from data sources on the data development platform through the metadata center and schema service, saving the step of manual programming. This not only simplifies the data preview process but also improves real-time processing efficiency.

[0150] In an optional implementation, the method may further include:

[0151] If a logical table corresponding to a data source is detected, the target schema information corresponding to the logical table is obtained from the schema service.

[0152] As an example only, the real-time computing platform can search locally for the existence of a logical table corresponding to a data source. If a logical table corresponding to a data source is detected, it means that a user has already registered that logical table in the real-time computing platform, thus eliminating the need for repeated registration of the logical table and allowing the retrieval of the logical table's name. The Schema service here can be the Schema service in the application environment diagram above. The Schema service can store the mapping relationship between logical table names and Schema information, which can be achieved by registering the corresponding logical table and Schema information in the Schema service. Then, based on the logical table name, the target Schema information corresponding to that logical table name is retrieved from the Schema service.

[0153] In the above embodiment, by detecting whether there is a logical table corresponding to the data source, when the logical table is detected, the second schema information is obtained directly using the logical table. The current user does not need to register the logical table corresponding to the data source again, which can reduce the amount of data processing computation, improve the efficiency of real-time calculation, and realize the sharing of the registered table among multiple users.

[0154] Figure 5 This is a partial flowchart illustrating a data processing method according to an exemplary embodiment, such as... Figure 5 As shown, the data processing method can be applied to electronic devices or data processing devices. Taking the electronic device as an example and the data processing system and the real-time computing platform as the Flink SQL platform in the above implementation environment diagram, the method includes the following steps.

[0155] In step S501, the real-time computing platform searches for the existence of a logical table corresponding to the data source. If it exists, the real-time computing platform uses the logical table and executes part of the steps in step S503 and subsequent steps; if it does not exist, step S502 is executed.

[0156] In step S502, the metadata center is searched to see if there is a registered data source corresponding to the data source. If there is, the registered data source and the data source are bound together on the real-time computing platform, and the metadata information corresponding to the data source is determined based on the basic data information of the registered data source and the data definition file corresponding to the data source. If there is no data source, the data source information corresponding to the data source is obtained, and the data source is registered in the metadata center based on the data source information. The metadata information corresponding to the data source is determined based on the basic data information of the registered data source and the data definition file corresponding to the data source.

[0157] In step S503, the real-time computing platform analyzes the basic information of the data according to the data definition file, obtains the first data table structure schema information corresponding to the data source, and obtains the data table structure adjustment information of the first schema information (such as deletion of fields, modification of field names, modification of field descriptions, setting as primary keys, etc.). Based on the first schema information (such as (field 1, type 1), (field 2, type 2), and (field 3, type 3)) and the data table structure adjustment information (such as the adjustment operation information for deleting field 2), the platform obtains the second schema information (such as (field 1, type 1) and (field 3, type 3)). The second schema information includes field names and field types. Based on the field names and field types in the second schema information, the data source is registered as a logical table. The schema information of the data source, such as field names and field types, is displayed, and registration information is sent to the schema service. The registration information includes the schema information of the data source.

[0158] The Schema service periodically or upon request from the real-time computing platform retrieves change information of the data source from the metadata center (such as adjustments to field names and / or field types) and dynamically updates the data source schema information.

[0159] In step S504, the user writes SQL statement information based on the field names and field types of the data source displayed by the real-time computing platform, without needing to create the data source and table using DDL. The real-time computing platform receives the SQL statement information.

[0160] In step S505, the real-time computing platform parses the SQL statement information to obtain the logical table name and the information to be operated, and obtains the corresponding schema information from the schema service based on the logical table name.

[0161] In step S506, the real-time computing platform creates a Flink Table corresponding to the data source based on the schema information; when the Flink job runs, it dynamically parses the data source data according to the information to be operated and the corresponding Flink Table to complete the real-time computation.

[0162] The specific methods and beneficial effects of the methods in this embodiment have been described in detail in the foregoing embodiments of the methods, and will not be elaborated here.

[0163] Figure 6 This is a block diagram of a data processing apparatus according to an exemplary embodiment. (Refer to...) Figure 6 The device includes:

[0164] The first acquisition module 610 is configured to acquire the metadata of the data source when it is detected that no logical table corresponding to the data source exists. The metadata includes basic data information and a data definition file. The data definition file is used to define the serialization and deserialization mechanism of the data source.

[0165] The second acquisition module 620 is configured to perform analysis of the basic information of the data according to the data definition file, and acquire the first data table structure schema information corresponding to the data source.

[0166] The third acquisition module 630 is configured to acquire data table structure adjustment information for the first schema information, and obtain second schema information based on the first schema information and the data table structure adjustment information; the second schema information includes field names and field types;

[0167] Registration module 640 is configured to register the data source as a logical table based on the field names and field types in the second schema information;

[0168] The parsing module 650 is configured to execute and obtain the structured query language SQL statement information corresponding to the logical table, and parse the SQL statement information to obtain the logical table name and the information to be operated, and obtain the target schema information corresponding to the logical table name;

[0169] The first processing module 660 is configured to create a target data table matching the current task environment based on the target schema information;

[0170] The second processing module 670 is configured to process the target data table according to the information to be operated.

[0171] As an optional implementation, the first acquisition module includes:

[0172] The first acquisition submodule is configured to retrieve the data source information corresponding to the data source if no registered data source matching the data source exists.

[0173] The second acquisition submodule is configured to register the data source based on the data source information and acquire the basic data information corresponding to the registered data source.

[0174] The first determining submodule is configured to retrieve the data definition file corresponding to the data source, and determine the metadata of the data source based on the data definition file and the basic data information.

[0175] As an optional implementation, the first acquisition module includes:

[0176] The binding submodule is configured to bind the registered data source to the data source if a registered data source matching the data source exists.

[0177] The third acquisition submodule is configured to acquire basic data information of the registered data source.

[0178] The second determining submodule is configured to retrieve the data definition file corresponding to the data source, and determine the metadata corresponding to the registered data source based on the data definition file and the basic data information, wherein the metadata corresponding to the registered data source is the metadata of the data source.

[0179] As an optional implementation, the device further includes:

[0180] The registration module is configured to register the second schema information and the corresponding logical table name of the data source to the schema service.

[0181] The update module is configured to update the second schema information in the schema service based on the real-time schema information corresponding to the data source stored in the metadata center.

[0182] As an optional implementation, the parsing module includes:

[0183] The display submodule is configured to display the second schema information corresponding to the logical table;

[0184] The fourth acquisition submodule is configured to retrieve structured language SQL statements based on the displayed second schema information.

[0185] As an optional implementation, the second processing module includes:

[0186] The first processing submodule is configured to dynamically parse the data from the data source based on the field names and field types corresponding to the target data table;

[0187] The second processing submodule is configured to execute the processing instructions indicated by the operation information on the parsed data.

[0188] As an optional implementation, the device further includes:

[0189] The fourth acquisition module is configured to retrieve the target schema information corresponding to the logical table from the schema service when a logical table corresponding to the data source is detected.

[0190] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0191] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (Refer to...) Figure 7 The electronic device includes a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the steps of any of the data processing methods described above.

[0192] The electronic device can be a terminal, a server, or a similar computing device. Taking a server as an example... Figure 7 This is a block diagram of an electronic device 700 for a data processing method, illustrated according to an exemplary embodiment. The electronic device 700 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 710 (CPUs 710 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 730 for storing data, and one or more storage media 720 (e.g., one or more mass storage devices) for storing application programs 723 or data 722. The memory 730 and storage media 720 may be temporary or persistent storage. The program stored in the storage media 720 may include one or more modules, each module including a series of instruction operations on the electronic device. Furthermore, the CPU 710 may be configured to communicate with the storage media 720 and execute the series of instruction operations in the storage media 720 on the electronic device 700.

[0193] Electronic device 700 may also include one or more power supplies 760, one or more wired or wireless network interfaces 750, one or more input / output interfaces 740, and / or one or more operating systems 721, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0194] The input / output interface 740 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 700. In one example, the input / output interface 740 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In an exemplary embodiment, the input / output interface 740 can be a radio frequency (RF) module for wireless communication with the Internet.

[0195] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, electronic device 700 may also include... Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown.

[0196] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of electronic device 700 to perform the above method. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0197] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the data processing method provided in any of the above embodiments.

[0198] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0199] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: When it is detected that there is no logical table corresponding to the data source, the metadata of the data source is obtained, including basic data information and data definition file; The data definition file is used to define the serialization and deserialization mechanism of the data source; the data definition file is determined according to the schema type of the data source in the metadata, the data source includes the target data source without a data table structure to be used, and the fields in the target data source are byte arrays that have been serialized with data values; The basic information of the data is analyzed according to the data definition file to obtain the first schema information corresponding to the data source; Obtain data table structure adjustment information for the first schema information, and based on the first schema information and the data table structure adjustment information, obtain second schema information; the second schema information includes field names and field types; Based on the field names and field types in the second schema information, the data source is registered as a logical table; Obtain the Structured Query Language (SQL) statement information corresponding to the logical table, parse the SQL statement information to obtain the logical table name and the information to be operated, and obtain the target schema information corresponding to the logical table name; Based on the target schema information, the data source is created as a target data table that matches the current task environment; The target data table is processed according to the information to be operated.

2. The method according to claim 1, characterized in that, The acquisition of metadata from the data source includes: If no registered data source matches the data source, obtain the data source information corresponding to the data source. Register the data source according to the data source information, and obtain the basic data information corresponding to the registered data source; Obtain the data definition file corresponding to the data source, and determine the metadata of the data source based on the data definition file and the basic data information.

3. The method according to claim 1, characterized in that, The acquisition of metadata from the data source includes: If a registered data source that matches the data source exists, bind the registered data source to the data source; Obtain basic data information from the registered data source; Obtain the data definition file corresponding to the data source, and determine the metadata corresponding to the registered data source based on the data definition file and the basic data information. The metadata corresponding to the registered data source is the metadata of the data source.

4. The method according to any one of claims 1-3, characterized in that, After the step of registering the data source as a logical table based on the field names and field types in the second schema information, the method further includes: Register the second schema information and corresponding logical table name of the data source to the schema service; The second schema information in the schema service is updated based on the real-time schema information corresponding to the data source stored in the metadata center.

5. The method according to claim 4, characterized in that, The step of obtaining the Structured Query Language (SQL) statement information corresponding to the logical table includes: Display the second schema information corresponding to the logical table; Obtain information about structured language SQL statements written based on the displayed second schema information.

6. The method according to any one of claims 1-3, characterized in that, The step of processing the target data table based on the information to be operated includes: Dynamically parse the data from the data source based on the field names and field types corresponding to the target data table; The processing instructions indicated by the pending operation information are executed on the parsed data.

7. The method according to any one of claims 1-3, characterized in that, The method further includes: If a logical table corresponding to a data source is detected, the target schema information corresponding to the logical table is obtained from the schema service.

8. A data processing apparatus, characterized in that, include: The first acquisition module is configured to acquire the metadata of the data source when a logical table corresponding to the data source is detected to be nonexistent. The metadata includes basic data information and a data definition file. The data definition file is used to define the serialization and deserialization mechanism of the data source. The data definition file is determined according to the schema type of the data source in the metadata. The data source includes a target data source without a data table structure that needs to be used. The fields in the target data source are byte arrays that have been serialized with data values. The second acquisition module is configured to perform analysis of the basic information of the data according to the data definition file, and acquire the first schema information corresponding to the data source; The third acquisition module is configured to acquire data table structure adjustment information for the first schema information, and obtain second schema information based on the first schema information and the data table structure adjustment information; the second schema information includes field names and field types; The registration module is configured to register the data source as a logical table based on the field names and field types in the second schema information. The parsing module is configured to execute and obtain the Structured Query Language (SQL) statement information corresponding to the logical table, and parse the SQL statement information to obtain the logical table name and the information to be operated, and obtain the target schema information corresponding to the logical table name; The first processing module is configured to create a target data table that matches the current task environment based on the target schema information; The second processing module is configured to process the target data table according to the information to be operated.

9. The apparatus according to claim 8, characterized in that, The first acquisition module includes: The first acquisition submodule is configured to retrieve the data source information corresponding to the data source if no registered data source matching the data source exists. The second acquisition submodule is configured to register the data source based on the data source information and acquire the basic data information corresponding to the registered data source. The first determining submodule is configured to retrieve the data definition file corresponding to the data source, and determine the metadata of the data source based on the data definition file and the basic data information.

10. The apparatus according to claim 8, characterized in that, The first acquisition module includes: The binding submodule is configured to bind the registered data source to the data source if a registered data source matching the data source exists. The third acquisition submodule is configured to acquire basic data information of the registered data source. The second determining submodule is configured to retrieve the data definition file corresponding to the data source, and determine the metadata corresponding to the registered data source based on the data definition file and the basic data information, wherein the metadata corresponding to the registered data source is the metadata of the data source.

11. The apparatus according to any one of claims 8-10, characterized in that, The device further includes: The registration module is configured to register the second schema information and the corresponding logical table name of the data source to the schema service. The update module is configured to update the second schema information in the schema service based on the real-time schema information corresponding to the data source stored in the metadata center.

12. The apparatus according to claim 11, characterized in that, The parsing module includes: The display submodule is configured to display the second schema information corresponding to the logical table; The fourth acquisition submodule is configured to retrieve structured language SQL statements based on the displayed second schema information.

13. The apparatus according to any one of claims 8-10, characterized in that, The second processing module includes: The first processing submodule is configured to dynamically parse the data from the data source based on the field names and field types corresponding to the target data table; The second processing submodule is configured to execute the processing instructions indicated by the operation information on the parsed data.

14. The apparatus according to any one of claims 8-10, characterized in that, The device further includes: The fourth acquisition module is configured to retrieve the target schema information corresponding to the logical table from the schema service when a logical table corresponding to the data source is detected.

15. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data processing method as described in any one of claims 1 to 7.

16. A storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the data processing method as described in any one of claims 1 to 7.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1 to 7.

Images