Data writing method and device and electronic equipment

Abstract

The invention discloses a data writing method and device and electronic equipment. The method comprises the following steps: acquiring data from a read data source to obtain initial data; determining a target operation corresponding to the initial data; a database expression function corresponding to the target operation is called, the database expression function is obtained by converting a target abstract syntax tree, the target abstract syntax tree is obtained by adjusting an initial abstract syntax tree, and the initial abstract syntax tree is obtained by analyzing a structured query statement corresponding to the target operation; processing the initial data by using a database expression function to obtain target data; and writing the target data into the result data source. According to the method and the device, the technical problem of low data writing efficiency caused by frequent analysis of the structured query grammar when the structured query language is used for data writing in related technologies is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data writing method, device and electronic equipment. Background Art

[0002] Data migration is extremely important in the field of modern data management and information technology. In the process of migrating data from the original data source to the target data source, the method currently used for data processing and writing is to use structured query language for filtering and conversion. However, this method requires frequent parsing of structured query language, which leads to the technical problem of low data writing efficiency.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a data writing method, device and electronic device to at least solve the technical problem in the related art that when using structured query language to write data, it is necessary to frequently parse structured query syntax, resulting in low data writing efficiency.

[0005] According to one aspect of an embodiment of the present invention, a data writing method is provided, comprising: acquiring data from a read data source to obtain initial data; determining a target operation corresponding to the initial data; calling a database expression function corresponding to the target operation, wherein the database expression function is obtained by transforming a target abstract syntax tree, the target abstract syntax tree is obtained by adjusting an initial abstract syntax tree, and the initial abstract syntax tree is obtained by parsing a structured query statement corresponding to the target operation; using the database expression function to process the initial data to obtain target data; and writing the target data into a result data source.

[0006] Optionally, before calling the database expression function corresponding to the target operation, it also includes: determining whether there is an existence result of the database expression function corresponding to the target operation; when the existence result is that the database expression function does not exist, determining a structured query statement corresponding to the target operation; parsing the structured query statement to obtain the initial abstract syntax tree; adjusting the initial abstract syntax tree to obtain a target abstract syntax tree; transforming the target abstract syntax tree to obtain the database expression function.

[0007] Optionally, the initial abstract syntax tree is adjusted to obtain a target abstract syntax tree, including: determining the character type corresponding to the initial data, and the operation operator corresponding to the target operation; determining a first original processing method corresponding to the character type, and a second original processing method corresponding to the operation operator in the initial abstract syntax tree; determining a first simplified processing method corresponding to the character type based on the first original processing method, and determining a second simplified processing method corresponding to the operation operator based on the second original processing method, wherein the first simplified processing method is a method whose corresponding amount of calculation is less than the amount of calculation corresponding to the first original processing method, and the second simplified processing method is a method whose corresponding amount of calculation is less than the amount of calculation corresponding to the second original processing method; using the first simplified processing method to replace the first original processing method, and using the second simplified processing method to replace the second original processing method to obtain the target abstract syntax tree.

[0008] Optionally, before calling the database expression function corresponding to the target operation, it also includes: determining a data reading application scenario; determining multiple related operations corresponding to the data reading application scenario; and constructing database expression functions corresponding to the multiple related operations respectively.

[0009] Optionally, determining the target operation corresponding to the initial data includes: determining the target operation corresponding to the initial data according to a storage data type and a data reading application scenario corresponding to the result data source.

[0010] Optionally, writing the target data into the result data source includes: determining a usage probability of the database expression function; and deleting the database expression function when the usage probability is lower than a predetermined threshold.

[0011] Optionally, writing the target data into a result data source includes: writing the target data into a cache queue; reading the target data from the cache queue and writing the target data into the result data source.

[0012] According to one aspect of an embodiment of the present invention, there is provided a data writing device, comprising: a reading module, used to obtain data from a read data source to obtain initial data; a determining module, used to determine a target operation corresponding to the initial data; a calling module, used to call a database expression function corresponding to the target operation, wherein the database expression function is obtained by transforming a target abstract syntax tree, the target abstract syntax tree is obtained by adjusting an initial abstract syntax tree, and the initial abstract syntax tree is obtained by parsing a structured query statement corresponding to the target operation; a processing module, used to process the initial data using the database expression function to obtain target data; and a writing module, used to write the target data into a result data source.

[0013] According to one aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; and a storage repository for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement any of the above data writing methods.

[0014] According to one aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute any of the above data writing methods.

[0015] In an embodiment of the present invention, data is obtained from a read data source to obtain initial data; a target operation corresponding to the initial data is determined; a database expression function corresponding to the target operation is called, wherein the database expression function is obtained by converting a target abstract syntax tree, the target abstract syntax tree is obtained after adjusting the initial abstract syntax tree, and the initial abstract syntax tree is obtained by parsing a structured query statement corresponding to the target operation; the initial data is processed using the database expression function to obtain the target data; and the target data is written into the result data source. By using the database expression function to process the initial data to obtain the target data, the effect of writing the target data into the result data source is achieved. Since the target abstract syntax tree is obtained by greatly simplifying the syntax structure of the initial abstract syntax tree to reduce the amount of calculation, the database expression function is obtained by converting the target abstract syntax tree, and can be directly called to process the initial data. Therefore, when processing the initial data, according to the target operation, the corresponding database expression function is directly called, which can reduce the frequent parsing of the structured query syntax and directly obtain the target result, thereby solving the technical problem of low data writing efficiency caused by the need to frequently parse the structured query syntax when writing data using the structured query language in the related technology.

[0016] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0017] SQL: Structured Query Language, referred to as SQL, is a special programming language, a database query and programming language used to access data and query, update and manage relational database systems.

[0018] DataX: A distributed data synchronization tool that can synchronize multi-source data to different target databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 is a flow chart of a data writing method according to an embodiment of the present invention;

[0021] Figure 2 is an operation flow chart of DataX provided by an optional embodiment of the present invention;

[0022] Figure 3 is a flowchart of the operation of DataX after the processing engine is introduced, provided by an optional embodiment of the present invention;

[0023] Figure 4 It is a structured query language syntax tree node logic diagram provided by an optional implementation mode of the present invention;

[0024] Figure 5 It is a flowchart of the operation of the database expression engine provided by an optional embodiment of the present invention;

[0025] Figure 6 is a life cycle diagram of a conversion module provided in an optional embodiment of the present invention;

[0026] Figure 7 is a structural block diagram of a data writing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Example 1

[0030] According to an embodiment of the present invention, an embodiment of a data writing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] Figure 1 is a flow chart of a data writing method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0032] Step S102, acquiring data from a read data source to obtain initial data.

[0033] In step S102 provided in the present application, data is acquired from a read data source to obtain initial data.

[0034] This involves reading the data source, which refers to the original storage location of the data. This can be any system that stores or generates data, such as relational databases, file systems, data warehouses, big data platforms (such as Hadoop HDFS), IoT sensors, etc.

[0035] Among them, the initial data is involved. The initial data refers to the data directly read from the data source without any modification or processing. This data usually contains all the original fields and values, and is the basis for subsequent data processing, cleaning, conversion, analysis, etc.

[0036] In this step, data is read from the specified data source, which can be any system or location containing data. This process extracts data from its storage location and forms the starting point for data processing. It is the most basic step in the data processing and data synchronization process, and provides a data foundation for subsequent data conversion, cleaning, and analysis.

[0037] It should be noted that when reading data from the data source, the system can standardize the data, such as data type conversion, data format unification, data encoding conversion, etc., to ensure that the acquired data meets the data standards for subsequent data processing and data synchronization operations, thereby improving the efficiency and accuracy of data processing. Data can be pre-processed, such as data filtering, data cleaning, data quality inspection, etc., to ensure that the acquired data is clean, effective, and high-quality, providing an excellent data foundation for subsequent data processing and data synchronization operations.

[0038] Step S104, determining the target operation corresponding to the initial data.

[0039] In step S104 provided in the present application, a target operation corresponding to the initial data is determined.

[0040] Among them, target operations are involved. Target operations refer to specific tasks that require processing or conversion of initial data, such as data cleaning, format conversion, data fusion, data migration to different databases, data analysis, machine learning training, etc. Target operations define how to process data to achieve the expected business goals or data quality requirements.

[0041] In this step, after obtaining the initial data, the specific operations that need to be performed on the data are determined according to specific needs. Through this step, the target operation is determined, which helps to plan and design the data processing flow in the future, ensuring that each step moves towards the final goal and avoiding unnecessary data processing work.

[0042] Step S106, calling a database expression function corresponding to the target operation, wherein the database expression function is obtained by transforming the target abstract syntax tree, the target abstract syntax tree is obtained by adjusting the initial abstract syntax tree, and the initial abstract syntax tree is obtained by parsing the structured query statement corresponding to the target operation.

[0043] In step S106 provided in the present application, a database expression function corresponding to the target operation is called.

[0044] Among them, the database expression function is involved. The database expression function refers to a function used to perform specific data operations, which is converted from the target abstract syntax tree.

[0045] Among them, the target abstract syntax tree is involved. The target abstract syntax tree refers to the abstract syntax tree that is adjusted based on the initial abstract syntax tree according to specific requirements or optimization strategies.

[0046] Among them, the initial abstract syntax tree is involved. The initial abstract syntax tree refers to the abstract syntax tree directly parsed from the structured query statement. It contains all the syntax elements and structural information of the query statement corresponding to the target operation of the initial data.

[0047] Among them, structured query statements are involved. Structured query statements refer to a standard language used to manage databases, used for operations such as data query, data update, data definition and data control.

[0048] In this step, the database expression function corresponding to the target operation is directly called. The database expression function is based on the standard query language of structured query statements. First, it is parsed to obtain the initial abstract syntax tree. Then, it is adjusted again according to specific needs or optimization strategies to obtain the target abstract syntax tree. Finally, the database expression function obtained by the transformation of the target abstract syntax tree has a variety of data processing functions, such as data screening, aggregation, connection, conversion, etc. By benchmarking the target operation, the appropriate database expression function is selected, and then specific data operations can be performed directly in the database.

[0049] Through this step, the database expression function is directly called to optimize the execution efficiency of the query statement, reduce the burden on the database, and improve the data processing speed. At the same time, the database expression function also supports dynamic parameterization, so that the function can be reused to process different data sets or data under different conditions, providing a high degree of adaptability and scalability.

[0050] It should be noted that the database expression function can be adjusted according to the characteristics of the target database to ensure the compatibility and correct execution of query statements on different database systems.

[0051] Step S108, using the database expression function to process the initial data to obtain target data.

[0052] In step S108 provided in the present application, the initial data is processed using a database expression function to obtain target data.

[0053] Among them, target data is involved. Target data refers to data that meets specific requirements or formats after being processed by database expression functions. It is the end point of the data processing process and is used to be stored in the target database.

[0054] In this step, the database expression function is called to operate on the initial data read from the read data source. These operations are determined by the target operation and may include data filtering to remove irrelevant or non-compliant data; data format conversion to meet the requirements of the target database or data processing system; data aggregation to perform statistical analysis or generate summary data, etc. Through these operations, the target data that meets specific requirements or formats is obtained and can be stored in the result data source.

[0055] Through this step, the initial data is processed using the database expression function, which can adopt the simplest processing mechanism to perform complex data conversion and cleaning operations, thereby ensuring the quality and integrity of the target data while improving the speed and efficiency of data processing.

[0056] It should be noted that the progress and status of data processing need to be monitored in real time, including data reading speed, data processing speed, data writing speed, data integrity check, etc., to ensure the reliability of data processing. At the same time, an exception handling mechanism needs to be designed to deal with various abnormal situations in the data processing process, such as data source failure, data processing failure, data writing failure, etc., to ensure the integrity and consistency of data processing.

[0057] Step S110, writing the target data into the result data source.

[0058] In step S110 provided in the present application, the target data is written into the result data source.

[0059] Among them, the result data source is involved, which refers to the system or location where the target data will be stored or written at the end of the data processing process. For example, relational databases, data warehouses, file systems, big data platforms, cloud storage services, etc.

[0060] In this step, the obtained target data is written to the result data source so that other systems or applications can access and use them.

[0061] It should be noted that the data writing process includes data consistency checks and data integrity verification to ensure that the data written to the data source is consistent with the expectations before processing, avoiding data errors or data contamination. When writing data to the result data source, data encryption, access control and other means can be used to ensure data security and privacy protection to prevent unauthorized access or tampering during transmission or storage. At the same time, in order to save storage space or enhance data security, compression operations are usually performed before data is written, which can significantly reduce storage usage.

[0062] Through the above steps S102-S110, data is obtained from the read data source to obtain initial data; the target operation corresponding to the initial data is determined; the database expression function corresponding to the target operation is called, wherein the database expression function is obtained by converting the target abstract syntax tree, the target abstract syntax tree is obtained after adjusting the initial abstract syntax tree, and the initial abstract syntax tree is obtained by parsing the structured query statement corresponding to the target operation; the initial data is processed using the database expression function to obtain the target data; and the target data is written into the result data source. By using the database expression function to process the initial data to obtain the target data, the effect of writing the target data into the result data source is achieved. Since the target abstract syntax tree is obtained by greatly simplifying the syntax structure of the initial abstract syntax tree to reduce the amount of calculation, the database expression function is obtained by converting the target abstract syntax tree, and can be directly called to process the initial data. Therefore, when processing the initial data, according to the target operation, the corresponding database expression function is directly called, which can reduce the frequent parsing of the structured query syntax and directly obtain the target result, thereby solving the technical problem of low data writing efficiency caused by the need to frequently parse the structured query syntax when writing data using the structured query language in the related technology.

[0063] As an optional embodiment, before calling the database expression function corresponding to the target operation, it also includes: determining whether there is an existence result of the database expression function corresponding to the target operation; when the existence result is that the database expression function does not exist, determining the structured query statement corresponding to the target operation; parsing the structured query statement to obtain an initial abstract syntax tree; adjusting the initial abstract syntax tree to obtain a target abstract syntax tree; transforming the target abstract syntax tree to obtain a database expression function.

[0064] In this embodiment, specific steps of determining a database expression function when no database expression function exists are described.

[0065] Among them, the existence result is involved, and the existence result refers to the result of determining whether there is a database expression function that can perform the target operation.

[0066] In this step, before calling the database expression function corresponding to the target operation, first determine whether there is a corresponding database expression function for the target operation. If there is no corresponding database expression function, it is necessary to first determine the structured query statement corresponding to the target operation. The target operation corresponding to the initial data is written in the structured query statement. Secondly, these structured query statements are parsed to construct a tree representation of the grammatical structure of the query language to obtain an initial abstract syntax tree. After that, according to the characteristics and requirements of the target database system, the initial abstract syntax tree is optimized and adjusted to obtain the target abstract syntax tree. Finally, the target abstract syntax tree is adjusted and converted into a database expression function. The database expression function can be directly executed by the database system to implement the processing logic of the target operation.

[0067] Through this step, firstly, the existence of the database expression function is checked to avoid repeatedly building the same function, thereby improving the reusability of the code and the efficiency of the system; secondly, when there is no database expression function, the database expression function is determined based on the structured query statement to improve the execution efficiency of the target operation and the data processing speed.

[0068] As an optional embodiment, the initial abstract syntax tree is adjusted to obtain a target abstract syntax tree, including: determining the character type corresponding to the initial data, and the operation operator corresponding to the target operation; determining a first original processing method corresponding to the character type and a second original processing method corresponding to the operation operator in the initial abstract syntax tree; determining a first simplified processing method corresponding to the character type based on the first original processing method, and determining a second simplified processing method corresponding to the operation operator based on the second original processing method, wherein the first simplified processing method is a method whose corresponding amount of calculation is less than the amount of calculation corresponding to the first original processing method, and the second simplified processing method is a method whose corresponding amount of calculation is less than the amount of calculation corresponding to the second original processing method; using the first simplified processing method to replace the first original processing method, and using the second simplified processing method to replace the second original processing method to obtain the target abstract syntax tree.

[0069] In this embodiment, specific steps of adjusting the initial abstract syntax tree to obtain the target abstract syntax tree are described.

[0070] Among them, the character type is involved. The character type refers to the text data type of the data field, such as variable character type (VARCHAR), fixed-length character type (CHAR), text type (TEXT), etc. The choice of character type affects the storage and processing of data, as well as query performance.

[0071] Among them, operation operators are involved. Operation operators refer to symbols or keywords used to perform specific data operations, that is, algorithms, formulas, etc. used to perform target operations.

[0072] Among them, the first original processing method is involved, and the first original processing method refers to a method of performing logical processing on the character type corresponding to the initial data in the initial abstract syntax tree.

[0073] Among them, the second original processing method is involved. The second original processing method refers to the method of performing logical processing on the operation operator corresponding to the target operation in the initial abstract syntax tree.

[0074] Among them, the first simplified processing method is involved. The first simplified processing method refers to optimizing the first original processing method, minimizing the amount of calculation and resource consumption, and performing logical processing on the character type corresponding to the initial data.

[0075] Among them, the second simplified processing method is involved. The second simplified processing method refers to optimizing the second original processing method, minimizing the amount of calculation and resource consumption, and performing logical processing on the operation operator corresponding to the target operation.

[0076] In this step, first, determine the character type of the initial data and the operation operator corresponding to the target operation. The determination of the character type and the operation operator can provide basic information for subsequent processing; secondly, based on the character type and the operation operator, determine the first original processing method corresponding to the character type in the initial abstract syntax tree and the second original processing method corresponding to the operation operator. The calculation process of these two original processing methods is relatively complex and the calculation cost is high. Therefore, continue to optimize and adjust these two original processing methods to reduce the amount of calculation and obtain a more efficient first simplified processing method corresponding to the character type and a second simplified processing method corresponding to the operation operator. Finally, use the first simplified processing method to replace the first original processing method, and use the second simplified processing method to replace the second original processing method. Based on the first simplified processing method and the second simplified processing method, determine the target abstract syntax tree.

[0077] Through this step, the first simplified processing method corresponding to the character type and the second simplified processing method corresponding to the operation operator are determined, which simplifies the subsequent data processing method, reduces the amount of calculation during data processing, improves the execution efficiency of data processing and query, and reduces resource consumption and cost.

[0078] It should be noted that for simplified processing of character types, you can convert character types to numeric types or other more efficient data types, for example, convert strings to integers or enumeration types to reduce the amount of calculation. For simplified processing of operation operators, you can cache intermediate results to reduce repeated calculations and improve data processing efficiency.

[0079] As an optional embodiment, before calling the database expression function corresponding to the target operation, it also includes: determining a data reading application scenario; determining multiple related operations corresponding to the data reading application scenario; and constructing database expression functions corresponding to the multiple related operations respectively.

[0080] In this embodiment, specific steps for determining a database expression function according to a data reading application scenario are described.

[0081] Among them, data reading application scenarios are involved. Data reading application scenarios refer to specific scenarios corresponding to specific business or functional requirements that need to be completed after reading data from the data source. For example, report generation, data analysis, real-time monitoring, data integration and other scenarios, each of which corresponds to different data requirements and processing logic.

[0082] Among them, related operations are involved. Related operations refer to a series of data processing operations that need to be performed for specific data reading application scenarios in order to complete specific data processing or business logic. For example, data screening, data conversion, data aggregation, data connection, etc. These operations together constitute a complete data processing flow.

[0083] In this step, first, based on the understanding of the specific operation tasks, the specific scenarios and purposes for reading data are identified, and the data reading application scenarios are determined. Secondly, multiple related operations that need to be performed on the initial data in this application scenario are determined. Finally, database expression functions corresponding to these related operations are constructed.

[0084] Through this step, multiple related operations in the data reading application scenario are constructed as database expression functions. These functions can be reused in different application scenarios and data processing processes, reducing the repeated development of code, improving development efficiency, and achieving high reusability of database expression functions. At the same time, the construction and management of database expression functions make the data processing logic clearer and independent, and easy to maintain and upgrade. When business needs or data processing rules change, that is, the data reading application scenario changes, only the corresponding database expression function needs to be modified or added, without large-scale modifications to the entire data processing process.

[0085] It should be noted that in order to improve data processing efficiency, the system will cache the generated database expression functions. When there are similar application scenarios or data processing requirements, the functions in the cache can be called directly to avoid repeated construction and optimization processes. For different data reading application scenarios, database expression functions can be dynamically called and combined to support the implementation of complex data processing logic and business logic. This enables the system to flexibly adapt to changing business needs and data processing scenarios. In addition, each database expression function can include security controls and data access permissions to ensure that only authorized operations can be performed, enhancing system security and avoiding data leakage and unauthorized access.

[0086] As an optional embodiment, determining the target operation corresponding to the initial data includes: determining the target operation corresponding to the initial data according to the storage data type corresponding to the result data source and the data reading application scenario.

[0087] In this embodiment, specific steps of determining the target operation corresponding to the initial data are described.

[0088] Among them, the storage data type is involved. The storage data type refers to the format or type used to store data in the result data source, such as numeric type, string type, date and time type, complex data type, etc. The choice of storage data type directly affects the efficiency of data storage, query performance, and data availability.

[0089] In this step, first, according to the storage data type of the result data source, identify the preliminary operations that need to be performed when processing the data. For example, if the result data source needs to store numeric data, and the initial data is in string format, then the target operation may include data type conversion, converting the string to a numeric type. Secondly, consider the characteristics of the data reading application scenario and determine the specific operations. For example, in the report generation scenario, data aggregation and statistics may be required; in the real-time monitoring scenario, data screening and real-time updates may be required; in the data analysis scenario, data cleaning and preprocessing may be required. Finally, based on the preliminary operations and specific operations, the target operation corresponding to the initial data is finally determined.

[0090] Through this step, the target operation corresponding to the initial data is determined based on the storage data type corresponding to the result data source and the data reading application scenario, ensuring the accuracy and reliability of the data processing flow and avoiding data conversion errors and deviations in data processing logic.

[0091] As an optional embodiment, writing the target data into the result data source includes: determining the usage probability of the database expression function; and deleting the database expression function when the usage probability is lower than a predetermined threshold.

[0092] In this embodiment, specific steps for processing a database expression function when writing target data into a result data source are described.

[0093] Among them, the usage probability is involved, which refers to the frequency with which the database expression function is called within a specific time.

[0094] Among them, a predetermined threshold is involved, and the predetermined threshold refers to an indicator set to determine whether the probability of using a database expression function meets system requirements.

[0095] In this step, first, determine the frequency with which the database expression function has been called in recent time, that is, the usage probability. Secondly, compare the usage probability with a predetermined threshold to determine whether the usage probability of the database expression function is lower than the predetermined threshold. Finally, if the usage probability of the database expression function is lower than the predetermined threshold, it is regarded as a low-utilization function and the database expression function is deleted.

[0096] Through this step, low-utilization database expression functions are deleted, the number of database expression functions in memory is reduced, the system maintenance cost is reduced, and the overall performance of the system is improved.

[0097] As an optional embodiment, writing the target data into the result data source includes: writing the target data into a cache queue, reading the target data from the cache queue, and writing the target data into the result data source.

[0098] In this embodiment, specific steps of writing target data into a result data source are described.

[0099] Among them, the cache queue is involved. The cache queue refers to a data structure used to temporarily store data. It is used to buffer data before the write operation to achieve batch writing of data, improve writing efficiency, and reduce database load. The cache queue can be a data structure in memory or a temporary file on disk.

[0100] In this step, after obtaining the target data, first, the target data is written into a cache queue, which can temporarily store the target data. Later, when the target data in the cache queue reaches a certain amount or time interval, the target data can be read from the cache queue and written into the result data source in batches.

[0101] Through this step, by using the cache queue, the write operations are concentrated in a short time, so as to realize batch writing of data and reduce the number and time of write operations. On the one hand, it can reduce the continuous writing pressure on the database, reduce the database load, improve the stability and response speed of the database, and on the other hand, it can improve the data writing efficiency.

[0102] It should be noted that the size and write strategy of the cache queue can be adjusted according to system resources, network conditions and business needs. For example, for high-concurrency business scenarios, a larger cache queue and a shorter write time interval can be set to improve write efficiency and system stability. In the process of writing to the cache queue, the target data can be compressed and encoded to reduce the overhead of data transmission and storage and improve the speed and efficiency of data processing. In the process of using the cache queue for data writing, design exception handling and fault-tolerant mechanisms to deal with abnormal situations such as network failures, system interruptions, data loss, etc., to ensure data integrity and consistency, as well as system stability.

[0103] Based on the above embodiments and optional embodiments, an optional implementation is provided, which is described in detail below.

[0104] During the process of migrating data from the original data source to the target data source, DataX is used to process and write data. Figure 2 is an operation flow chart of DataX provided by an optional implementation mode of the present invention, such as Figure 2 As shown in the figure, in the process of data processing, the transformation module (Transformer) is a very important component, which can transform, clean and process data to meet different data requirements.

[0105] However, in actual applications, users may perform more complex query operations on data, and in this case, a SQL query engine needs to be introduced to support these operations. Figure 3 is a flowchart of the operation of DataX after the processing engine is introduced, as provided in an optional embodiment of the present invention. Figure 3 As shown, the query engine can retrieve the corresponding data from the data storage according to the user's query request and return it to the user. In this way, the user can query and analyze the data more conveniently, thereby better meeting the needs of data analysis and mining.

[0106] During data processing, the SQL query engine will analyze the logic, conditions, table names, column names, etc. SQL parsing is required for each record filtering. Figure 4 is a structured query language syntax tree node logic diagram provided by an optional implementation manner of the present invention, such as Figure 4 As shown in the figure, multiple nodes (SQLnode) will be created during the entire data processing process. The frequent creation of these nodes will greatly occupy memory and affect the efficiency of data processing.

[0107] For this situation, an optional implementation of the present invention provides a DataX data filtering and processing method based on a SQL query engine, and introduces a database expression engine (DB expression engine) to optimize it.

[0108] DB expression engine is a lightweight calculation engine. Figure 5 is a flowchart of the operation of the database expression engine provided by an optional embodiment of the present invention, such as Figure 5 As shown, the DB expression engine can compile user-defined expressions into efficient machine codes, thereby improving the efficiency of data processing.

[0109] The steps of the optional implementation mode of the present invention are explained in detail below.

[0110] S1. Obtain data from the read data source to obtain initial data.

[0111] The Reader module of DataX is responsible for reading data from a data source, such as a database, file system, or other data storage.

[0112] S2. Determine the target operation corresponding to the initial data.

[0113] Determine the target operation corresponding to the initial data based on the storage data type corresponding to the result data source and the data reading application scenario.

[0114] S3. Call the database expression function corresponding to the target operation.

[0115] A1. Determine whether there is a database expression function corresponding to the target operation.

[0116] A2. When the result is that the database expression function does not exist, determine the database expression function.

[0117] Figure 6 is a life cycle diagram of a conversion module provided in an optional embodiment of the present invention, such as Figure 6 As shown, when the result is that there is no database expression function, the structured query language is parsed, the plan is optimized, and the database expression (DBExprFunction) execution plan is generated, and finally the database expression function is obtained. The specific steps are as follows.

[0118] B1. Determine a structured query statement corresponding to the target operation; parse the structured query statement to obtain an initial abstract syntax tree.

[0119] If the user defines SQL query or data conversion logic, DataX will first parse these SQL statements and generate an abstract syntax tree (AST, the same as the initial abstract syntax tree mentioned above). AST is a tree-like data structure that can clearly represent the logical structure of SQL statements, including table names, field names, filter conditions, arithmetic operations, etc.

[0120] B2. Determine the character type corresponding to the initial data and the operation operator corresponding to the target operation.

[0121] B3. Determine a first original processing method corresponding to the character type and a second original processing method corresponding to the operation operator in the initial abstract syntax tree.

[0122] B4. Determine a first simplified processing method corresponding to the character type according to the first original processing method, and determine a second simplified processing method corresponding to the operation operator according to the second original processing method.

[0123] This step is usually carried out from the following aspects.

[0124] First, database systems usually collect and maintain statistical information about data, including the number of rows in a table, the distribution of columns, the use of indexes, etc. This information can be used to estimate the cost of query processing methods, thereby selecting a more efficient and simplified processing method. For example, if a conditional expression involves a column that is highly selective (meaning that the values ​​of the column are widely distributed and each value corresponds to a small number of rows), then choose to use an index instead of a full table scan.

[0125] The second is to check the index structure of the table and select the most appropriate index to use according to the query conditions. If the conditional expression can be quickly located using the index, adjust the query plan and change the data access method from full table scan to index access, thereby greatly improving the query speed.

[0126] The third is to evaluate the execution costs of different operators, such as sorting, joining, grouping, etc. Try to find the simplest way to simplify the processing with the lowest cost. For example, you may choose to execute a lower-cost operator (such as filtering) first, and then execute a higher-cost operator (such as joining) to avoid unnecessary data processing.

[0127] Fourth, push down the operation operators and push down some processing operations (such as filtering and projection) to the nodes where the data is located for execution, thereby reducing the transmission of data in the network and improving the overall processing efficiency.

[0128] Fifth, consider the available resources and parallel processing capabilities in the system and try to generate simplified processing methods that can fully utilize these resources. For operations that can be executed in parallel, generate corresponding parallel processing methods to reduce the overall execution time.

[0129] Sixth, analyze the characteristics of functions and expressions in the operator, and try to optimize the execution path, such as avoiding unnecessary function calls or adjusting function calls to a more appropriate execution stage.

[0130] In addition, check the semantics and logic of the simplified processing method to ensure that the optimized simplified processing method can still return query results correctly.

[0131] B5. Use the first simplified processing method to replace the first original processing method, and use the second simplified processing method to replace the second original processing method to obtain a target abstract syntax tree.

[0132] B6. Transform the target abstract syntax tree to obtain the database expression function.

[0133] The optimized AST is compiled into a DBExprFunction, which not only represents the logic of the SQL query, but also contains all the information and instructions required to execute the query.

[0134] During the compilation process, the DB expression engine converts the logical operations in the AST into specific memory operation instructions, which can be executed directly in the memory without repeatedly accessing disk or network resources.

[0135] When the same expression needs to be executed multiple times, the DB expression engine compiles the expression into an internal data structure and caches it. The next time the same expression needs to be executed, the DB expression engine directly uses the cached data structure, thus avoiding re-parsing and building the expression and improving computing efficiency.

[0136] A3. Determine the data reading application scenario, determine the multiple related operations corresponding to the data reading application scenario, and determine the database expression functions corresponding to the multiple related operations. Select and call the appropriate DBExprFunction according to the operation in the SQL statement and the data type of the initial data.

[0137] S4. Use the database expression function to process the initial data to obtain the target data.

[0138] The transformation module (Transformer) of DataX will call the DBExprFunction that matches the data type to filter, transform or otherwise process the data.

[0139] Because DBExprFunction is precompiled and optimized, it can be efficiently executed in memory without re-parsing and optimizing SQL statements every time data is processed, which greatly improves the speed and efficiency of data processing.

[0140] S5. Write the target data into the result data source.

[0141] Write the target data into the cache queue; read the target data from the cache queue and write it into the result data source. The processed data is transmitted to the writing module through the queue and finally written into the target database or data storage.

[0142] S6. Determine the usage probability of the database expression function, and delete the database expression function when the usage probability is lower than a predetermined threshold.

[0143] The DB expression engine also optimizes the data structure in the cache to improve the cache hit rate and efficiency, which can effectively reduce the consumption of computing resources and improve the efficiency of data processing.

[0144] Through the above optional implementation, at least the following beneficial effects can be achieved:

[0145] (1) Expand DataX custom data processing by referencing the SQL query engine. Implement more data conversion and processing functions, introduce the DB expression engine, optimize the parsing calculation of data processing, reduce the creation of memory objects, improve data processing efficiency, and provide users with a general SQL query language to perform intermediate data processing on DataX tasks, thereby improving development efficiency;

[0146] (2) By using the cache queue, the data writing process is optimized, the consistency and integrity of the data is ensured, the risk of directly writing to the database is reduced, and the system management is more convenient and secure;

[0147] (3) By managing the usage probability and life cycle of database expression functions, low-utilization functions can be automatically deleted, reducing unnecessary code and configuration in the system and simplifying the management and maintenance of database expression functions.

[0148] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0150] Example 2

[0151] According to an embodiment of the present invention, a device for implementing the above data writing method is also provided. Figure 7 is a structural block diagram of a data writing device according to an embodiment of the present invention. Figure 7 As shown, the device includes: a reading module 702, a determining module 704, a calling module 706, a processing module 708 and a writing module 710. The device is described in detail below.

[0152] A reading module 702 is used to obtain data from a read data source to obtain initial data; a determining module 704 is connected to the reading module 702 and is used to determine a target operation corresponding to the initial data; a calling module 706 is connected to the determining module 704 and is used to call a database expression function corresponding to the target operation, wherein the database expression function is obtained by transforming a target abstract syntax tree, the target abstract syntax tree is obtained by adjusting an initial abstract syntax tree, and the initial abstract syntax tree is obtained by parsing a structured query statement corresponding to the target operation; a processing module 708 is connected to the calling module 706 and is used to process the initial data using the database expression function to obtain target data; a writing module 710 is connected to the processing module 708 and is used to write the target data into a result data source.

[0153] It should be noted here that the above-mentioned reading module 702, determination module 704, calling module 706, processing module 708 and writing module 710 correspond to steps S102 to S110 in implementing the data writing method, and the instances and application scenarios implemented by the multiple modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0154] Example 3

[0155] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; and a storage repository for storing processor-executable instructions, wherein the processor is configured to execute instructions to implement any of the above data writing methods.

[0156] Example 4

[0157] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute any of the above data writing methods.

[0158] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0159] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0162] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0164] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data writing method, characterized in that: include: Get data from the read data source to obtain initial data; Determining a target operation corresponding to the initial data; Retrieve a database expression function corresponding to the target operation, wherein the database expression function is obtained by transforming a target abstract syntax tree, the target abstract syntax tree is obtained by adjusting an initial abstract syntax tree, and the initial abstract syntax tree is obtained by parsing a structured query statement corresponding to the target operation; Processing the initial data using the database expression function to obtain target data; The target data is written into the result data source.

2. The method according to claim 1, characterized in that Before calling the database expression function corresponding to the target operation, the method further includes: Determine whether there is a result of a database expression function corresponding to the target operation; In a case where the existence result is that the database expression function does not exist, determining a structured query statement corresponding to the target operation; Parsing the structured query statement to obtain the initial abstract syntax tree; Adjust the initial abstract syntax tree to obtain the target abstract syntax tree; The target abstract syntax tree is transformed to obtain the database expression function.

3. The method according to claim 2, characterized in that Adjust the initial abstract syntax tree to obtain the target abstract syntax tree, including: Determine the character type corresponding to the initial data and the operation operator corresponding to the target operation; Determine, in the initial abstract syntax tree, a first original processing mode corresponding to the character type and a second original processing mode corresponding to the operation operator; Determine a first simplified processing method corresponding to the character type according to the first original processing method, and determine a second simplified processing method corresponding to the operation operator according to the second original processing method, wherein the first simplified processing method is a method whose corresponding calculation amount is less than the calculation amount corresponding to the first original processing method, and the second simplified processing method is a method whose corresponding calculation amount is less than the calculation amount corresponding to the second original processing method; The first simplified processing method is used to replace the first original processing method, and the second simplified processing method is used to replace the second original processing method to obtain the target abstract syntax tree.

4. The method according to claim 1, characterized in that: Before calling the database expression function corresponding to the target operation, the method further includes: Determine the data reading application scenario; Determine a plurality of related operations corresponding to the data reading application scenario; Construct database expression functions corresponding to the multiple related operations respectively.

5. The method according to claim 1, characterized in that Determine the target operation corresponding to the initial data, including: The target operation corresponding to the initial data is determined according to the storage data type corresponding to the result data source and the data reading application scenario.

6. The method according to claim 1, characterized in that Writing the target data into the result data source includes: Determining a usage probability of the database expression function; When the usage probability is lower than a predetermined threshold, the database expression function is deleted.

7. The method according to any one of claims 1 to 6, characterized in that: Writing the target data into the result data source includes: Writing the target data into a cache queue; The target data is read from the cache queue and written into the result data source.

8. A data writing device, characterized in that: include: The reading module is used to obtain data from the reading data source to obtain initial data; A determination module, used to determine a target operation corresponding to the initial data; A calling module, used to call a database expression function corresponding to the target operation, wherein the database expression function is obtained by transforming a target abstract syntax tree, the target abstract syntax tree is obtained by adjusting an initial abstract syntax tree, and the initial abstract syntax tree is obtained by parsing a structured query statement corresponding to the target operation; A processing module, used to process the initial data using the database expression function to obtain target data; A writing module is used to write the target data into a result data source.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data writing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data writing method as claimed in any one of claims 1 to 7.

Images