Data processing method and device, equipment and storage medium

Abstract

The application relates to a data processing method, device and equipment and a storage medium. The method comprises the following steps: analyzing a target script, identifying a first identifier, a second identifier and a target query statement in the target script, the first identifier corresponding to a source data warehouse, and the second identifier corresponding to a target data warehouse; determining access configuration information of the source data warehouse in preset warehouse registration data according to the first identifier, and executing a query operation in the source data warehouse through the access configuration information and the target query statement to obtain a first query result; and storing the first query result into the target data warehouse according to the second identifier. The method can reduce the complexity of script programming, thereby improving the development efficiency of the script.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the development of technology, data warehouse technology is becoming increasingly diversified. Different vendors and types of data warehouses (such as relational databases, MPP databases, data lakes, etc.) have significant differences in access protocols, SQL dialects, and connection parameters.

[0003] In practical applications, the current common practice is to hard-code the data warehouse access address, authentication information, and data source-specific syntax adaptation logic directly into the script, which are the access configuration data.

[0004] However, because different data warehouses require different access configuration methods, when a script involves multiple heterogeneous data warehouses, technical personnel must repeatedly write the corresponding access configuration code for each data warehouse within the script. This approach makes script writing complex and results in low development efficiency. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium that can reduce the complexity of script writing and thus improve script development efficiency.

[0006] In a first aspect, this application provides a data processing method, the method comprising: Parse the target script and identify the first identifier, the second identifier, and the target query statement in the target script. The first identifier corresponds to the source data warehouse, and the second identifier corresponds to the target data warehouse. Based on the first identifier, the access configuration information of the source data warehouse is determined in the preset warehouse registration data, and a query operation is performed in the source data warehouse using the access configuration information and the target query statement to obtain the first query result; The first query result is stored in the target data warehouse according to the second identifier.

[0007] Secondly, this application provides a data processing apparatus, the apparatus comprising: The parsing unit is used to parse the target script and identify the first identifier, the second identifier, and the target query statement in the target script. The first identifier corresponds to the source data warehouse, and the second identifier corresponds to the target data warehouse. The determining unit is configured to determine the access configuration information of the source data warehouse in the preset warehouse registration data according to the first identifier, and perform a query operation in the source data warehouse through the access configuration information and the target query statement to obtain a first query result; A storage unit is used to store the first query result into the target data warehouse according to the second identifier.

[0008] Thirdly, this application provides a data processing device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to: Parse the target script and identify the first identifier, the second identifier, and the target query statement in the target script. The first identifier corresponds to the source data warehouse, and the second identifier corresponds to the target data warehouse. Based on the first identifier, the access configuration information of the source data warehouse is determined in the preset warehouse registration data, and a query operation is performed in the source data warehouse using the access configuration information and the target query statement to obtain the first query result; The first query result is stored in the target data warehouse according to the second identifier.

[0009] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described data processing method.

[0010] Compared with the prior art, the technical solution provided in this application has the following advantages: In this application embodiment, the target script is parsed to identify the first identifier, the second identifier, and the target query statement in the target script. The first identifier corresponds to the source data warehouse, and the second identifier corresponds to the target data warehouse. Based on the first identifier, the access configuration information of the source data warehouse is determined from the preset warehouse registration data. A query operation is performed in the source data warehouse using the access configuration information and the target query statement to obtain a first query result. The first query result is stored in the target data warehouse based on the second identifier. By introducing the first identifier of the source data warehouse and the second identifier of the target data warehouse, this application allows users to declare only the identifier and standard query statement when writing the target script, without needing to concern themselves with the physical implementation details such as the connection information, access protocol, and storage format of the underlying data warehouse. This reduces the complexity of script writing and improves script development efficiency. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0014] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 2 This is a schematic flowchart of a data processing apparatus provided in an embodiment of this application; Figure 3 This is a schematic diagram of a data processing device provided in an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0017] With the development of technology, data warehouse technology is becoming increasingly diversified. Different vendors and types of data warehouses (such as relational databases, MPP databases, data lakes, etc.) have significant differences in access protocols, SQL dialects, and connection parameters.

[0018] In practice, the current common practice is to hard-code the data warehouse access address, authentication information, and data source-specific syntax adaptation logic directly into the script.

[0019] However, because different data warehouses require different access configuration methods, when a script involves multiple heterogeneous data warehouses, technical personnel must repeatedly write the corresponding access configuration code for each data warehouse in the target script. This approach makes script writing complex and results in low script development efficiency.

[0020] To address the aforementioned problems, embodiments of this application provide a data processing method that reduces the complexity of script writing, thereby improving script development efficiency. Figure 1 As shown, the specific steps include: Step 101: Parse the target script and identify the first identifier, the second identifier, and the target query statement in the target script.

[0021] The target script is a standardized instruction file used to implement data flow and governance between data warehouses. When the target script is an execution script in MySQL, it is an SQL governance script. The target script includes at least a first identifier corresponding to the source data warehouse, a second identifier corresponding to the target data warehouse, and the target query statement.

[0022] The aforementioned identifier is a resource abstraction identifier indicating a data warehouse. It includes a prefix and a data warehouse identifier. The prefix indicates that what follows is the data warehouse identifier, which is a unique identity identifier registered by the corresponding data warehouse in the data governance system. For example, the data warehouse identifier can be the instance ID of the data warehouse, the hash value of the connection string, or a custom alias. For example, in $db_s3_0001, $ is the prefix, and db_s3_0001 is the data warehouse identifier. In this application, the resource abstraction identifier corresponding to the source data warehouse is called the first identifier, meaning that the first identifier includes the data warehouse identifier of the source data warehouse, which is used to provide raw data for query governance. The resource abstraction identifier corresponding to the target data warehouse is called the second identifier, meaning that the second identifier includes the data warehouse identifier of the target data warehouse, which is used to receive the governed results for further analysis. For example, the first identifier is $starrocks_file_001, where starrocks_file_001 is the data warehouse identifier corresponding to the source data warehouse, and the second identifier is $db_s3_0001, where db_s3_0001 is the data warehouse identifier corresponding to the target data warehouse.

[0023] All the data warehouses mentioned above are pre-registered with the data governance system. This means that technical personnel need to log in to the data governance system and register both the source and target data warehouses. Taking $starrocks_file_001 as the first identifier and $db_s3_0001 as the second identifier, a source data warehouse registered as StarRocks has the corresponding data warehouse identifier $starrocks_file_001, recording its JDBC URL, username and password, table structure, and SQL dialect. Similarly, a target data warehouse registered as Amazon S3 has the corresponding data warehouse identifier $db_s3_0001, recording its Bucket information, AccessKey, Parquet write format, and path rules. Finally, the metadata registry in the data governance system stores the binding relationship between the two types of resource identifiers and physical storage, completing the data initialization process.

[0024] The target query is a query statement using a specified language, such as a query statement conforming to the SQL standard, used to perform operations such as extraction, filtering, and aggregation on the source data. In practice, the target query and the first identifier appear in the same SQL statement. For example, the statement `CREATE TABLE tb_temp AS SELECT cphm, img_path FROM $starrocks_file_001 WHERE day = #beginDate` includes the target query (filtering out the data corresponding to `cphm` and `img`) and the first identifier `$starrocks_file_001`.

[0025] Parsing refers to the process of performing lexical, syntactic, and semantic analysis on a script using a syntax analyzer (or parser), ultimately generating an Abstract Syntax Tree (AST). It is used to construct a structured AST from unstructured script text and extract key information elements, such as primary identifiers, secondary identifiers, and the target query statement (the specific query statement), by traversing the AST. These elements are then used to perform corresponding data governance tasks. The target query statement is used for query operations performed in the source data warehouse, such as filtering operations.

[0026] In this step, technicians write and submit the target script through the visual interface of the data governance system. After receiving the target script submitted by the user, the data governance system uses a preset parser to parse the target script and obtain an abstract syntax tree (AST). Based on the AST, the system identifies the first identifier, the second identifier, and the target query statement in the target script.

[0027] In the actual target script, the parser may also process some custom syntax, such as $db_s3_0001, $starrocks_file_001, @if conditional control, and #beginDate variable substitution, as described in this application. These extended keywords will also be recognized by lexical / syntactic analysis and incorporated into the abstract syntax tree, ultimately performing corresponding operations according to the context semantics.

[0028] In addition, after obtaining the target script, the data governance system can perform an integrity check on the script to ensure that it contains the first identifier, the second identifier, and the target query statement. When the script is correctly formatted and all elements are complete, it is added to the task queue for processing, awaiting execution and parsing; when the script has format errors or missing elements, a format error message is returned to the user, and further execution of the script is rejected.

[0029] Step 103: Based on the first identifier, determine the access configuration information of the source data warehouse in the preset warehouse registration data, and perform a query operation in the source data warehouse using the access configuration information and the target query statement to obtain the first query result.

[0030] The access configuration information includes the data required by the data instruction system to access the data warehouse, such as format conversion strategies, statement conversion strategies, actual connection information, access credentials, and other information.

[0031] In this step, the data warehouse identifier included in the first identifier is determined, and a search and matching is performed in the data warehouse registry based on this data warehouse identifier. This registry maintains a mapping relationship between data warehouse identifiers and data warehouse access configuration information. Thus, based on the access configuration information of the source data warehouse found, the operation corresponding to the target query statement can be executed in the source data warehouse, thereby obtaining the first query result.

[0032] Step 103: Store the first query result in the target data warehouse according to the second identifier.

[0033] In this step, the data warehouse to be used, i.e., the target data warehouse, is determined based on the second identifier. Then, the results of the first query are written to the target data warehouse.

[0034] Writing the first query result to the target data warehouse typically involves two sub-processes: data format conversion and data storage. First, the system converts the first query result into the corresponding data format according to the storage format requirements of the target data warehouse. Then, the converted first query result is stored in the target data warehouse through its write interface. Alternatively, data format conversion may not be necessary, and the first query result can be directly stored in the target data warehouse.

[0035] The storage formats include Parquet, ORC, Avro, or plain text, and the data warehouse write interfaces include JDBC Batch Insert, HDFS put operation, and S3 putObject API interface.

[0036] In this embodiment, the target script is parsed to identify a first identifier, a second identifier, and a target query statement. The first identifier corresponds to the source data warehouse, and the second identifier corresponds to the target data warehouse. Based on the first identifier, the access configuration information of the source data warehouse is determined from preset warehouse registration data. Using the access configuration information and the target query statement, a query operation is performed in the source data warehouse to obtain a first query result. The first query result is then stored in the target data warehouse based on the second identifier. This application, by introducing a first identifier for the source data warehouse and a second identifier for the target data warehouse, allows users to declare only the identifiers and standard query statements when writing the target script, without needing to concern themselves with the physical implementation details of the underlying data warehouse, such as connection information, access protocols, and storage formats. This reduces the complexity of script writing and improves script development efficiency.

[0037] In this embodiment, by parsing the target script, a first identifier is identified, and a first repository identifier is determined from the first identifier. Then, a statement conversion strategy and access protocol corresponding to the first repository identifier are matched from preset repository registration data, serving as the first statement conversion strategy and first access protocol for subsequent use. Therefore, this embodiment provides a source data warehouse determination method, specifically including: determining the first statement conversion strategy and first access protocol of the source data warehouse from preset repository registration data based on the first identifier; and assembling the first statement conversion strategy and the first access protocol into the access configuration information.

[0038] The first statement transformation strategy adapts the syntax based on the syntax supported by the source data warehouse, while the first access protocol is determined based on the network interface type of the source data warehouse to ensure that query commands can be correctly received and executed. For example, the first access protocol might be the adapter corresponding to the source data warehouse. Network interface types include JDBC, REST API, or native client protocols.

[0039] The specific steps for determining the first statement conversion strategy and the first access protocol of the source data warehouse based on the first warehouse identifier in the preset warehouse registration data are as follows: The first warehouse identifier is determined from the first identifier. The warehouse registration data includes the warehouse identifier and its corresponding statement conversion strategy and access protocol. Therefore, the statement conversion strategy and access protocol corresponding to the first warehouse identifier can be found in the warehouse registration data and used as the first statement conversion strategy and the first access protocol, respectively.

[0040] Alternatively, when writing the first repository identifier, the repository type can be included, and the repository registration data can include the repository type and its corresponding statement conversion strategy and access protocol. In this way, the repository type of the source data warehouse can be obtained by identifying the first repository identifier; the statement conversion strategy and access protocol corresponding to the repository type can be obtained from the repository registration data, thus obtaining the first statement conversion strategy and the first access protocol.

[0041] In this embodiment, a first statement conversion strategy is used to convert the target query statement into a warehouse query statement compatible with the source data warehouse; then, a first access protocol is used to send the warehouse query statement to the source data warehouse. After receiving the warehouse query statement, the source data warehouse performs a query operation based on the warehouse query statement to obtain a first query result, and returns the first query result to the data governance system. Therefore, this embodiment provides a source data warehouse query method, the specific steps of which include: converting the target query statement into a warehouse query statement compatible with the source data warehouse using a first statement conversion strategy; sending the warehouse query statement to the source data warehouse through a first access protocol, instructing the source data warehouse to perform a query based on the warehouse query statement to obtain a first query result.

[0042] In this embodiment, a second identifier is used to determine the statement transformation strategy and access protocol corresponding to the target data warehouse. The statement transformation strategy is then used to transform the first query result to obtain the target query result. Finally, the access protocol is used to store the target query result in the target data warehouse. Therefore, this embodiment provides a method for writing to a target data warehouse, specifically including: determining the second access protocol corresponding to the target data warehouse from preset warehouse registration data based on the second identifier; and sending the first query result to the target data warehouse via the second access protocol, so that the target data warehouse stores the first query result.

[0043] The second access protocol is determined based on the network interface type of the source data warehouse, ensuring that query commands can be correctly received and executed. Network interface types include JDBC, REST API, or native client protocols.

[0044] The specific steps for determining the second access protocol corresponding to the target data warehouse based on the second warehouse identifier in the preset warehouse registration data are as follows: The second warehouse identifier is determined from the second identifier. The warehouse registration data includes the warehouse identifier and its corresponding access protocol; therefore, the access protocol corresponding to the second warehouse identifier can be found in the warehouse registration data and used as the second access protocol.

[0045] Alternatively, when writing the second repository identifier, the repository type can be included, and the repository registration data can include the repository type and its corresponding access protocol. In this way, the repository type of the target data warehouse can be obtained by identifying the second repository identifier; and the access protocol corresponding to that repository type can be obtained from the repository registration data, thus obtaining the second access protocol.

[0046] In actual storage, after obtaining the first query result, its format can be uniformly converted to an internal standard columnar batch (StandardColumnarBatch). Its schema includes field names, data types, and null value semantics, using an Apache Arrow compatible format as a general intermediate representation for cross-source data exchange. Then, since the current context is S3, `CREATE TABLE tb_temp` does not create a real table, but instead generates the physical path `s3: / / my-bucket / tb_temp / date=20260206 / `, and serializes the StandardColumnarBatch into a Parquet file and uploads it to that path. The mapping relationship between logical table names and physical paths is then recorded for subsequent use.

[0047] In this embodiment, the first calling statement is identified by parsing the target script, and then the corresponding registration function is searched from a pre-set set of registration functions to perform customized processing on the first query result, meeting diverse data governance needs. Therefore, this embodiment provides a data processing method based on registration functions, specifically including: using the first calling statement to search for the registration function corresponding to the first calling statement in a pre-set set of registration functions; in response to finding the registration function corresponding to the first calling statement, processing the first query result through the registration function to obtain a second query result; and storing the first query result and the second query result in the target data warehouse according to the second identifier.

[0048] The first invocation statement typically appears in the target script with a specific syntax, indicating the additional processing logic that needs to be performed on the query results; it includes at least a function name. The registered function set includes at least one registered function; for example, the registered function can be a UDF function.

[0049] In this step, during the parsing of the target script, a specific syntax form is identified, and its corresponding statement is determined as the first call statement. A registered function with the same name as the function included in the first call statement is searched in the registered function set; if found, this registered function is called to process the first query result and generate the second query result. The original first query result and the processed second query result are then stored together in the target data warehouse.

[0050] Alternatively, if not found, you can choose to ignore the call statement or terminate with an error, depending on the configuration.

[0051] For example, the first query statement `udf_get_charactor(img_path)` is parsed, and the function name `udf_get_charactor` is extracted. Based on the function name, the corresponding UDF definition is searched from the registry, including the input type (string) and output type (map).<string,string> The associated service endpoint is http: / / ai-service / extract_features. The vectorized UDF executor is started, the img_path column is read in batches and processed to obtain the second query result, which is then written to the corresponding path in S3.

[0052] In this embodiment, a first prefix is ​​identified by parsing the target script, and then the corresponding conditional control flow statement is determined. This conditional control flow statement is then used to logically process the first query result, satisfying the conditional data governance requirements in complex business scenarios. Therefore, this embodiment provides a data processing method based on conditional control flow, with specific steps including: parsing the target script and identifying the first prefix in the target script; determining the conditional control flow statement based on the first prefix in the target script; processing the first query result according to the conditional control flow statement to obtain a third query result; and storing the first query result and the third query result in the target data warehouse according to the second identifier.

[0053] The first prefix indicates that the following statement is a conditional control flow statement. It usually begins with a special symbol, such as @if, #if, or {{if, and is used to identify the conditional control logic embedded in the script.

[0054] In this step, upon parsing the target script and recognizing the first prefix, the subsequent statements are parsed as conditional control flow statements. The structure of a conditional control flow statement typically includes a conditional evaluation section and an operation to be performed when the condition is met, such as `@if (conditional expression) BEGIN ... END`. After recognizing the first prefix, the subsequent statements are extracted and executed. If the condition is true, the corresponding operation is executed, thereby processing the first query result and generating the third query result. Then, the target data warehouse is determined based on the second identifier, and the first and third query results are stored together in the target data warehouse.

[0055] For example, the above operations can be data filtering, aggregation calculation, anomaly marking, or alarm insertion.

[0056] For example, the parsing process retrieves control flow statements, which include subqueries and INSERT statements. The subquery is used in tb_final (i.e., the Parquet file on S3) to filter records where character_array['event']='accident', counting the number of duplicate license plates. Assuming the return value is 520, then 520-500=20>0, and the condition is true.

[0057] The following INSERT statement is then generated and executed: INSERT INTO tb_warningVALUES('20260206','Abnormal number of accident vehicles'). Since tb_warning does not explicitly specify a storage location, it inherits the current S3 context by default, and the data is written to the path s3: / / ... / tb_warning / . If the target is a relational database, the system will automatically switch adapters to execute native INSERT statements.

[0058] In this embodiment, the second prefix in the target query statement is identified by parsing the target script, and then the corresponding target variable is determined and replaced with an actual value to meet different data processing needs. Therefore, this embodiment provides a query statement processing method based on variable replacement, the specific steps of which include: identifying the second prefix in the target query statement and determining the target variable; obtaining the target value corresponding to the target variable; replacing the target variable included in the target query statement with the target value to obtain an updated target query statement; and performing a query operation in the source data warehouse using the access configuration information and the updated target query statement to obtain a first query result.

[0059] The second prefix is ​​used to identify variable placeholders embedded in the script. This second prefix is ​​usually a special symbol, such as #, $, or :. The target query may contain multiple such variables, for example, #beginDate in WHERE day = #beginDate.

[0060] In this step, upon parsing the target query statement, the second prefix is ​​identified, and it, along with the following string, is used as the target variable. Subsequently, the corresponding target value is searched for in a pre-defined variable mapping table or runtime context, for example, obtained from task parameters, configuration files, or upstream results. If found, the target variable is replaced with this target value, and the target query statement is updated to be used for further querying. If not found, the default value can be used or the process can terminate with an error, depending on the configuration.

[0061] For example, to get the current date as February 6, 2026, #beginDate will be automatically replaced with the string '20260206'.

[0062] like Figure 2 As shown, this application provides a data processing apparatus, which corresponds to the method embodiment, and specifically includes: Parsing unit 201 is used to parse the target script and identify the first identifier, the second identifier and the target query statement in the target script. The first identifier corresponds to the source data warehouse and the second identifier corresponds to the target data warehouse. The determining unit 202 is used to determine the access configuration information of the source data warehouse in the preset warehouse registration data according to the first identifier, and to perform a query operation in the source data warehouse through the access configuration information and the target query statement to obtain a first query result; Storage unit 203 is used to store the first query result to the target data warehouse according to the second identifier.

[0063] Optionally, the determining unit 202 is configured to: Based on the first identifier, determine the first statement conversion strategy and the first access protocol of the source data warehouse in the preset warehouse registration data; The first statement conversion strategy and the first access protocol are combined to form the access configuration information.

[0064] Optionally, the determining unit 202 is configured to: The target query statement is converted into a warehouse query statement compatible with the source data warehouse using the first statement conversion strategy. The first access protocol is used to send the warehouse query statement to the source data warehouse, instructing the source data warehouse to perform a query based on the warehouse query statement and obtain the first query result.

[0065] Optionally, the storage unit 203 is used for: Based on the second identifier, determine the second access protocol corresponding to the target data warehouse from the preset warehouse registration data; The first query result is sent to the target data warehouse via the second access protocol, so that the target data warehouse stores the first query result.

[0066] Optionally, the target script includes a first calling statement, and the determining unit 202 is used to: Using the first call statement, search for the registration function corresponding to the first call statement in the pre-set set of registration functions; In response to finding the registration function corresponding to the first call statement, the first query result is processed through the registration function to obtain the second query result; Based on the second identifier, the first query result and the second query result are stored in the target data warehouse.

[0067] Optionally, the target script includes a first prefix and conditional control flow statements including the first prefix, and the storage unit is used for: Based on the conditional control flow statement, the first query result is processed to obtain the third query result; Based on the second identifier, the first query result and the third query result are stored in the target data warehouse.

[0068] Optionally, the target query statement includes the second prefix and the target variable including the second prefix, and the determining unit 202 is used to: Identify the second prefix in the target query statement to determine the target variable; Obtain the target value corresponding to the target variable; The target variable included in the target query statement is replaced with the target value to obtain an updated target query statement; Using the access configuration information and the updated target query statement, a query operation is performed in the source data warehouse to obtain the first query result.

[0069] like Figure 3As shown in the figure, this application provides a data processing device, including a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. Memory 303 is used to store computer programs; In one embodiment of this application, when the processor 301 executes a program stored in the memory 303, it implements the data processing method provided in any of the foregoing method embodiments, including: Parse the target script and identify the first identifier, the second identifier, and the target query statement in the target script. The first identifier corresponds to the source data warehouse, and the second identifier corresponds to the target data warehouse. Based on the first identifier, the access configuration information of the source data warehouse is determined in the preset warehouse registration data, and a query operation is performed in the source data warehouse using the access configuration information and the target query statement to obtain the first query result; The first query result is stored in the target data warehouse according to the second identifier.

[0070] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps performed by the data processing method provided in any of the foregoing method embodiments.

[0071] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0073] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0074] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method, characterized in that, include: Parse the target script and identify the first identifier, the second identifier, and the target query statement in the target script. The first identifier corresponds to the source data warehouse, and the second identifier corresponds to the target data warehouse. Based on the first identifier, the access configuration information of the source data warehouse is determined in the preset warehouse registration data, and a query operation is performed in the source data warehouse using the access configuration information and the target query statement to obtain the first query result; The first query result is stored in the target data warehouse according to the second identifier.

2. The method according to claim 1, characterized in that, The step of determining the access configuration information of the source data warehouse from the preset warehouse registration data based on the first identifier includes: Based on the first identifier, determine the first statement conversion strategy and the first access protocol of the source data warehouse in the preset warehouse registration data; The first statement conversion strategy and the first access protocol are combined to form the access configuration information.

3. The method according to claim 2, characterized in that, The step of performing a query operation in the source data warehouse using the access configuration information and the target query statement to obtain a first query result includes: The target query statement is converted into a warehouse query statement compatible with the source data warehouse using the first statement conversion strategy. The first access protocol is used to send the warehouse query statement to the source data warehouse, instructing the source data warehouse to perform a query based on the warehouse query statement and obtain the first query result.

4. The method according to claim 1, characterized in that, The step of storing the first query result in the target data warehouse according to the second identifier includes: Based on the second identifier, determine the second access protocol corresponding to the target data warehouse from the preset warehouse registration data; The first query result is sent to the target data warehouse via the second access protocol, so that the target data warehouse stores the first query result.

5. The method according to claim 1, characterized in that, The target script includes a first calling statement, and storing the first query result in the target data warehouse according to the second identifier includes: Using the first call statement, search for the registration function corresponding to the first call statement in the pre-set set of registration functions; In response to finding the registration function corresponding to the first call statement, the first query result is processed through the registration function to obtain the second query result; Based on the second identifier, the first query result and the second query result are stored in the target data warehouse.

6. The method according to claim 1, characterized in that, The target script includes a first prefix and a conditional control flow statement including the first prefix. The step of storing the first query result in the target data warehouse according to the second identifier includes: Based on the conditional control flow statement, the first query result is processed to obtain the third query result; Based on the second identifier, the first query result and the third query result are stored in the target data warehouse.

7. The method according to claim 1, characterized in that, The target query statement includes the second prefix and a target variable including the second prefix. The step of performing a query operation in the source data warehouse using the access configuration information and the target query statement to obtain a first query result includes: Identify the second prefix in the target query statement to determine the target variable; Obtain the target value corresponding to the target variable; The target variable included in the target query statement is replaced with the target value to obtain an updated target query statement; Using the access configuration information and the updated target query statement, a query operation is performed in the source data warehouse to obtain the first query result.

8. A data processing apparatus, characterized in that, The device includes: The parsing unit is used to parse the target script and identify the first identifier, the second identifier, and the target query statement in the target script. The first identifier corresponds to the source data warehouse, and the second identifier corresponds to the target data warehouse. The determining unit is configured to determine the access configuration information of the source data warehouse in the preset warehouse registration data according to the first identifier, and perform a query operation in the source data warehouse through the access configuration information and the target query statement to obtain a first query result; A storage unit is used to store the first query result into the target data warehouse according to the second identifier.

9. A data processing device, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; At least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to implement the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 7.

Images