Method, apparatus, and storage medium for processing data
By receiving data transformation requests in the database system, reading and filtering data that conforms to quality monitoring rules, the problem of quality issues being transferred when converting source data tables to destination data tables is solved, achieving efficient and accurate data transformation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
- Filing Date
- 2022-12-28
- Publication Date
- 2026-07-31
AI Technical Summary
In a database system, when a source data table is converted into a destination data table, quality issues in the source data table are transferred to the destination data table, resulting in a decrease in data quality.
By receiving a data conversion request, data is read based on the identification information of the first data table, data that conforms to the first quality monitoring rule is identified, and the data is converted into second data and saved to the second data table. Data with quality problems is filtered using the first and second quality monitoring rules.
This effectively avoids transferring data with quality issues to the target data table, saves computing resources, and improves the accuracy and efficiency of data conversion.
Smart Images
Figure CN116049155B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computers, and in particular to a method, apparatus and storage medium for processing data. Background Technology
[0002] Database systems often consist of multiple tables, each used to store data. One table in a database system may be derived from another table within the same database system. For example, the first table might be derived from the second table, and the second table from the third table; the first, second, and third tables are different tables within the database system.
[0003] The data in the third data table has quality issues. The second data table, derived from the third, may also contain data with quality issues, and the first data table, derived from the second, may also contain data with quality issues. Since the second data table is derived from the third, it can be called the source data table, and the second data table can be called the destination data table. Similarly, the first data table is derived from the second; therefore, the second data table can be called the source data table, and the first data table can be called the destination data table.
[0004] Therefore, when converting a source data table to a destination data table, if there are quality issues in the source data table, these quality issues will be transferred to the destination data table during the conversion. Summary of the Invention
[0005] This application provides a method, apparatus, and storage medium for processing data to avoid transferring quality problems from the source data table to the destination data table when converting a source data table into a destination data table. The technical solution is as follows:
[0006] In a first aspect, this application provides a method for processing data. In this method, a data transformation request is received, the request including identification information of a first data table, identification information of a second data table, and a first quality monitoring rule. Based on the identification information of the first data table, multiple sets of first data stored in the first data table are read. First data that conforms to the first quality monitoring rule is determined from among the multiple sets of first data. The first data conforming to the first quality monitoring rule is converted into at least one set of second data. Based on the identification information of the second data table, the at least one set of second data is saved to the second data table.
[0007] Since the data conversion request includes a first quality monitoring rule, based on the identification information of the first data table, multiple first data items stored in the first data table are read. The first data items that conform to the first quality monitoring rule are identified, and these first data items are converted into at least one second data item. By converting the first data items that conform to the first quality monitoring rule, first data items with quality problems in the first data table can be excluded, while first data items without quality problems in the first data table can be converted. Then, based on the identification information of the second data table, the at least one second data item is saved to the second data table, thereby preventing first data items with quality problems in the first data table from being transferred to the second data table during the conversion.
[0008] In one possible implementation, the data transformation request further includes a second quality monitoring rule to identify second data among the at least one set of second data that conforms to the second quality monitoring rule. Based on the identification information of the second data table, the second data that conforms to the second quality monitoring rule is saved to the second data table.
[0009] Since the at least one second data is obtained by converting multiple first data in the first data table, the second data that conforms to the second quality monitoring rule among the at least one second data is determined, and the second data that conforms to the second quality monitoring rule is saved to the second data table. This avoids saving second data with quality problems generated during the conversion process to the second data table.
[0010] In another possible implementation, the data transformation request includes a data read statement and a data write statement, wherein the data read statement includes a first quality monitoring rule and the data write statement includes a second quality monitoring rule.
[0011] If the data quality in the first data table is only checked, multiple first data points from the first data table need to be read separately, and these multiple first data points need to be checked using quality monitoring rules. Since in this application, the data read statement includes a first quality monitoring rule, and the data write statement includes a second quality monitoring rule, when reading data from the first data table based on the data read statement, the first quality monitoring rule is used to determine the first data points in the first data table that conform to the first quality monitoring rule. Before writing second data to the second data table based on the data write statement, the second quality monitoring rule is used to determine the second data points in the second data to be saved that conform to the second quality monitoring rule. During the process of converting the first data table to the second data table, since multiple first data points need to be read from the first data table, and data conforming to the quality monitoring rules needs to be determined from these multiple first data points, it is not necessary to read all the multiple first data points in the first data table separately when checking the data quality, thus saving computational resources.
[0012] In another possible implementation, the data transformation request further includes first indication information, which indicates the position of the first quality monitoring rule within the data transformation request. Based on the position indicated by the first indication information, the first quality monitoring rule is read from the data transformation request. In this way, the first quality monitoring rule can be accurately obtained based on the first indication information.
[0013] Secondly, this application provides a data processing apparatus for performing the method in the first aspect or any possible implementation thereof. Specifically, the apparatus includes units for performing the method in the first aspect or any possible implementation thereof.
[0014] Thirdly, this application provides a data processing apparatus, including at least one processor and a memory, wherein the at least one processor is coupled to the memory to read and execute instructions in the memory to implement the method of the first aspect or any possible implementation of the first aspect.
[0015] Fourthly, this application provides a computer program product comprising a computer program stored in a computer-readable storage medium, wherein the computer program is loaded by a processor to implement the method described in the first aspect or any possible implementation thereof.
[0016] Fifthly, this application provides a computer-readable storage medium for storing a computer program, which is loaded by a processor to execute the method described in the first aspect or any possible implementation thereof.
[0017] In a sixth aspect, this application provides a chip including a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to retrieve and execute the computer instructions from the memory to perform the method described in the first aspect or any possible implementation thereof. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;
[0019] Figure 2 This is a schematic diagram of another network architecture provided in an embodiment of this application;
[0020] Figure 3 This is a flowchart of a data processing method provided in an embodiment of this application;
[0021] Figure 4 This is a schematic diagram of an input interface provided in an embodiment of this application;
[0022] Figure 5 This is a schematic diagram of a data processing device provided in an embodiment of this application;
[0023] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;
[0024] Figure 7 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;
[0025] Figure 8 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation
[0026] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.
[0027] See Figure 1 This application provides a network architecture 100, which includes a computing engine 101 and a database system 102, with the computing engine 101 communicating with the database system 102.
[0028] Database system 102 includes multiple data tables, each used to store data.
[0029] The computing engine 101 can send a data access request to the database system 102, and the database system 102 can access the data tables in the database system 102 based on the data access request.
[0030] For example, computing engine 101 sends a data read request to database system 102, the data read request including identification information of the data table to be read. Database system 102 includes the data table to be read.
[0031] Database system 102 receives a data read request, and based on the identification information of the data table to be read included in the data read request, reads the data stored in the data table to be read in database system 102, and sends a data read response to computing engine 101, which includes the read data. Computing engine 101 receives the data read response.
[0032] For example, computing engine 101 sends a data write request to database system 102, the data write request including the identification information of the data table to be written and the data to be written. Database system 102 includes the data table to be written.
[0033] Database system 102 receives a data write request, and based on the identification information of the data table to be written included in the data write request, saves the data to be written to the data table to be written in database system 102, and sends a data write response to computing engine 101. The data write response includes indication information to indicate that the write was successful. Computing engine 101 receives the data write response.
[0034] For example, computing engine 101 sends a data transformation request to database system 102, the data transformation request including identification information of a first data table and identification information of a second data table. Database system 102 includes a first data table and a second data table.
[0035] Database system 102 receives a data transformation request, and based on the identification information of a first data table included in the data transformation request, reads multiple first data items stored in the first data table in database system 102; converts the multiple first data items into multiple second data items, and based on the identification information of a second data table, saves the multiple second data items into a second data table in database system 102; and sends a data transformation response to computing engine 101, the data transformation response including indication information for indicating successful transformation. Computing engine 101 receives the data transformation response.
[0036] In some embodiments, the number of computing engines 101 in the network architecture 100 is at least one. For example, see... Figure 2 The network architecture 100 includes two computing engines 101, namely Spark and Hive.
[0037] In some embodiments, the database system 102 includes at least one script. For any script, the script includes identification information of a source data table, identification information of a destination data table, and implementation code for implementing a conversion operation. The implementation code in the script is used to implement the conversion operation, which converts the source data table corresponding to the identification information of the source data table into the destination data table corresponding to the identification information of the destination data table.
[0038] For example, a database system includes a first script and a second script. The first script includes the identifier information of the source data table (which is the identifier information of the first data table) and the identifier information of the destination data table (which is the identifier information of the second data table). The first script also includes implementation code for performing a first transformation operation. The second script includes the identifier information of the source data table (which is the identifier information of the second data table) and the identifier information of the destination data table (which is the identifier information of the third data table). The second script also includes implementation code for performing a second transformation operation.
[0039] For any script in a database system, the script includes identification information for at least one source data table, and the script is used to instruct that the at least one source data table be converted into a destination data table.
[0040] In this way, when the database system 102 receives a data conversion request, it uses the identification information of the first data table included in the data conversion request as the identification information of the source data table, uses the identification information of the second data table included in the data conversion request as the identification information of the destination data table, obtains a first script including the identification information of the first data table and the identification information of the second data table, uses the implementation code included in the first script to convert multiple first data stored in the first data table into multiple second data, and saves the multiple second data into the second data table.
[0041] In this process, the first data table may contain first data with quality problems. When converting the first data into the second data, the second data may also have quality problems. In order to avoid saving the second data with quality problems into the second data table, the following embodiments can be used to avoid transferring the quality problems in the source data table to the destination data table when converting the source data table into the destination data table.
[0042] See Figure 3 This application provides a data processing method 300, which is applied to... Figure 1 or Figure 2 The network architecture 100 shown includes the following steps 301-305 of the method 300.
[0043] Step 301: The database system receives a data transformation request, which includes the identification information of the first data table, the identification information of the second data table, and the first quality monitoring rule.
[0044] The computing engine sends a data transformation request to the database system. This data transformation request is used to request the database system to convert multiple first data stored in the first data table into multiple second data, and save the multiple second data into the second data table.
[0045] In some embodiments, the data transformation request further includes first indication information, which indicates the position of the first quality monitoring rule in the data transformation request. Thus, after the database system receives the data transformation request, it identifies the first indication information from the data transformation request, determines the position of the first quality monitoring rule in the data transformation request based on the first indication information, and retrieves the first quality monitoring rule from the data transformation request based on that position.
[0046] In some embodiments, the data transformation request includes multiple data access statements, including data read statements and data write statements. The data read statement includes identification information of a first data table and a first quality monitoring rule, and the data write statement includes identification information of a second data table.
[0047] The data read statement is used to instruct the database system to read data from the first data table, and the data write statement is used to instruct the database system to write data to the second data table.
[0048] In some embodiments, the data read statement further includes first indication information, which indicates the position of the first quality monitoring rule in the data read statement.
[0049] In some embodiments, the multiple data access statements are Structured Query Language (SQL) statements. When a user needs to convert data in the first data table into data in the second data table, the user can input multiple SQL statements into the computing engine. The computing engine obtains the multiple SQL statements input by the user and sends a data conversion request to the database system. The data conversion request includes the multiple SQL statements.
[0050] For example, see Figure 4 The computing engine displays an input interface where users can enter multiple SQL statements, which the computing engine then reads from the input interface.
[0051] Suppose that the data transformation request includes the following SQL statements, and the data reading statements include three statements: The first statement is: "10", "id" AS "id" / *+validation not_null, unique* / . The second statement is: "10", "name" AS "name" / *+validation int[xx], id_card* / . The third statement is: SELECT "id", "name" FROM "table1".
[0052] The data read statement includes the identifier information "table1" for the first data table and the first instruction information "validation". The first data read statement includes the first quality control rule "not_null, unique", which is located after the first instruction information "validation". The first quality control rule "not_null, unique" indicates that the id column in the data table table1 cannot be empty and each piece of data in the id column is unique.
[0053] The second data read statement includes the first quality monitoring rule "int[xx], id_card". The first quality monitoring rule "int[xx], id_card" is located after the first indication information "validation". The first quality monitoring rule "int[xx], id_card" is used to indicate that each data in the name column of the data table table1 is an integer with a length of xx bits.
[0054] The data write statement includes JOIN(SELECT "id", "name" FROM "table2") AS "r1" ON "10.id" = "r1.id". This data write statement includes the identifier information for the second data table, "table2".
[0055] SELECT
[0056] "10", "id" AS "id" / *+validation not_null, unique* / ,
[0057] "10", "name" AS "name" / *+validation int[xx], id_card* / ,
[0058] "r1","id" AS "id1",
[0059] "r1", "name" AS "name1",
[0060] FROM
[0061] (SELECT "id","name" FROM "table1")AS "10"
[0062] JOIN (SELECT "id", "name" FROM "table2") AS "r1" ON "10.id" = "r1.id".
[0063] In some embodiments, the data write statement further includes a second quality monitoring rule. Optionally, the data write statement further includes second indication information, which indicates the position of the second quality monitoring rule within the data write statement.
[0064] The first quality monitoring rule and / or the second quality monitoring rule are used to monitor the integrity, validity, consistency, timeliness, and / or accuracy of the data.
[0065] Step 302: The database system reads multiple first data stored in the first data table based on the identification information of the first data table.
[0066] The data read statement includes the identification information of the first data table. The database system executes the data read statement so that the database system can read multiple first data stored in the first data table based on the identification information of the first data table.
[0067] In some embodiments, a data transformation request includes at least one data read statement, each data read statement being used to read at least one column of data from a first data table. For any data read statement, the data read statement includes identification information of at least one column of data in the first data table, and the database system executes the data read statement to read the at least one column of data from the first data table based on the identification information of the at least one column of data included in the data read statement.
[0068] Step 303: The database system determines the first data among the multiple first data that meets the first quality monitoring rule.
[0069] In some embodiments, for any data read statement, the data read statement includes identification information of at least one column of data in a first data table, first indication information, and a first quality monitoring rule. The database system identifies the first indication information from the data read statement and reads the first quality monitoring rule from the data read statement based on the first indication information.
[0070] In this way, after the database system reads the at least one column of data from the first data table based on the identification information of the at least one column of data included in the data read statement, it determines the first data in the at least one column of data that conforms to the first quality monitoring rule based on the first quality monitoring rule read.
[0071] For example, in the data read statement: "10", "id" AS "id" / *+validation not_null, unique* / , this data read statement includes the identifier information "id" of a column of data in the first data table "table1", the first indication information "validation", and the first quality control rule "not_null, unique". The database system identifies the first indication information "validation" from this data read statement and, based on the first indication information "validation", reads the first quality control rule "not_null, unique" from the data read statement.
[0072] Thus, the database system reads a column of data from the first data table "table1" based on the identifier information "id" included in the data read statement. Each row in this column contains an id. Based on the first quality control rule "not_null, unique", the system determines each id in the column that conforms to this rule. Specifically, for each id that conforms to the first quality control rule, all ids are distinct.
[0073] Step 304: The database system converts the first data that conforms to the first quality monitoring rule into at least one second data.
[0074] In some embodiments, the data transformation request further includes a data write statement, which in turn includes identification information of a second data table. The database system further includes at least one script. For any script, the script includes identification information of a source data table, identification information of a destination data table, and implementation code for implementing the transformation operation. The script is used to implement the transformation operation, converting the source data table corresponding to the identification information of the source data table into the destination data table corresponding to the identification information of the destination data table.
[0075] In step 304, the database system uses the identifier information of the first data table as the identifier information of the source data table and the identifier information of the second data table as the identifier information of the destination data table. It then obtains a script from the at least one script that includes the identifier information of the source data table and the identifier information of the destination data table. Based on the implementation code for performing the transformation operation included in the obtained script, the transformation operation is determined, and the first data conforming to the first quality monitoring rule is transformed into at least one second data item through this transformation operation.
[0076] In some embodiments, the conversion operation can be case conversion, sorting, and / or data concatenation, etc.
[0077] Step 305: The database system saves at least one piece of second data into the second data table based on the identification information of the second data table.
[0078] In some embodiments, the data conversion request further includes a data write statement, which also includes identification information of a second data table. The database system executes the data write statement to save at least one piece of second data into the second data table based on the identification information of the second data table included in the data write statement.
[0079] In some embodiments, the data writing statement further includes a second quality monitoring rule, which determines the second data among the at least one second data that conforms to the second quality monitoring rule; and saves the second data that conforms to the second quality monitoring rule to the second data table based on the identification information of the second data table.
[0080] Secondary data that does not conform to the second quality monitoring rule is data with quality problems generated during the conversion process. Therefore, secondary data that conforms to the second quality monitoring rule is saved to the secondary data table to avoid secondary data with quality problems in the secondary data table.
[0081] In some embodiments, the data write statement further includes second indication information. The database system identifies the second indication information from the data write statement, obtains the position of the second quality monitoring rule in the data write statement based on the second indication information, and obtains the second quality monitoring rule from the data write statement based on the obtained position. Then, the database system determines the second data that conforms to the second quality monitoring rule among the at least one second data; based on the identification information of the second data table, the second data that conforms to the second quality monitoring rule is saved to the second data table.
[0082] In this embodiment, the database system receives a data transformation request, which includes identification information of a first data table, identification information of a second data table, and a first quality monitoring rule. Based on the identification information of the first data table, multiple sets of first data stored in the first data table are read. First data that conforms to the first quality monitoring rule is determined from among the multiple sets of first data. The first data conforming to the first quality monitoring rule is converted into at least one set of second data. Because the first data conforming to the first quality monitoring rule is converted, first data with quality problems in the first data table can be excluded, and first data without quality problems in the first data table can be converted. Then, based on the identification information of the second data table, the database system saves the at least one set of second data into the second data table, thereby preventing first data with quality problems in the first data table from being transferred to the second data table during the conversion. Furthermore, before saving the at least one set of second data into the second data table, the database system also determines the second data that conforms to the second quality monitoring rule from among the at least one set of second data and saves the second data that conforms to the second quality monitoring rule into the second data table. In this process, the second data that does not conform to the second quality monitoring rule is data with quality issues generated during the conversion. Therefore, the database system identifies at least one set of second data that conforms to the second quality monitoring rule and saves this data to the second data table, thus avoiding the inclusion of quality-issued second data in the second data table. Furthermore, since the data conversion request includes the first quality monitoring rule, after receiving the request, the database system reads the first data from the first data table, identifies the first data that conforms to the first quality monitoring rule, and then converts this first data into second data. This process, by reading the first data from the first data table once, both the first quality monitoring rule determines that the first data in the first data table does not have quality issues, and the first data table is converted into the second data table, saving computational resources.
[0083] See Figure 5 This application provides a data processing apparatus 500, which is deployed in... Figure 1 or Figure 2 The database system 102 in the network architecture 100 shown, or the device 500 deployed in Figure 3 The method 300 shown is applied to a database system. The apparatus 500 includes:
[0084] The receiving unit 501 is used to receive a data conversion request, which includes the identification information of the first data table, the identification information of the second data table, and the first quality monitoring rule.
[0085] The processing unit 502 is used to read multiple first data stored in the first data table based on the identification information of the first data table;
[0086] Processing unit 502 is further configured to determine the first data among the plurality of first data that conforms to the first quality monitoring rule;
[0087] Processing unit 502 is further configured to convert first data that conforms to the first quality monitoring rule into at least one second data;
[0088] The processing unit 502 is also configured to save at least one piece of second data into the second data table based on the identification information of the second data table.
[0089] Optionally, for details on the implementation process of receiving unit 501 receiving data conversion requests, please refer to [link to relevant documentation]. Figure 3 The details of step 301 of method 300 shown will not be explained in detail here.
[0090] Optionally, for details of the implementation process of processing unit 502 reading multiple first data stored in the first data table, please refer to [link to relevant documentation]. Figure 3 The details of step 302 of method 300 shown will not be explained in detail here.
[0091] Optionally, the detailed implementation process of the processing unit 502 determining the first data that conforms to the first quality monitoring rule among the plurality of first data can be found in [link to documentation]. Figure 3 The details of step 303 of method 300 shown will not be explained in detail here.
[0092] Optionally, for details of the processing unit 502 converting the first data that conforms to the first quality monitoring rule into at least one second data, please refer to [link to relevant documentation]. Figure 3 The details of step 304 of method 300 shown will not be explained in detail here.
[0093] Optionally, for details of how processing unit 502 saves at least one piece of second data to the second data table, please refer to [link to relevant documentation]. Figure 3 The details of step 304 of method 300 shown will not be explained in detail here.
[0094] Optionally, the data transformation request also includes a second quality monitoring rule, which the processing unit 502 is used to:
[0095] Identify the second data among the at least one second data that conforms to the second quality monitoring rule;
[0096] Based on the identification information of the second data table, the second data that conforms to the second quality monitoring rules is saved into the second data table.
[0097] Optionally, the detailed implementation process of the processing unit 502 determining the second data that conforms to the second quality monitoring rule in the at least one second data can be found in [link to documentation]. Figure 3 The details of step 305 of method 300 shown will not be explained in detail here.
[0098] Optionally, for details on how processing unit 502 saves the second data that conforms to the second quality monitoring rules to the second data table, please refer to [link to relevant documentation]. Figure 3 The details of step 305 of method 300 shown will not be explained in detail here.
[0099] Optionally, the data transformation request includes a data read statement and a data write statement, wherein the data read statement includes a first quality monitoring rule and the data write statement includes a second quality monitoring rule.
[0100] Optionally, the data transformation request may further include first indication information, which indicates the location of the first quality monitoring rule within the data transformation request.
[0101] The processing unit 502 is also configured to read a first quality monitoring rule from the data conversion request based on the location indicated by the first indication information.
[0102] Optionally, the detailed implementation process of the processing unit 502 reading the first quality monitoring rule from the data transformation request can be found in [link to relevant documentation]. Figure 3 The details of step 301 of method 300 shown will not be explained in detail here.
[0103] Both the receiving unit 501 and the processing unit 502 can be implemented in software or in hardware. For example, the implementation of the processing unit 502 will be described below. Similarly, the implementation of the receiving unit 501 can be referenced to that of the processing unit 502.
[0104] As an example of a software functional unit, processing unit 502 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, processing unit 502 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.
[0105] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
[0106] As an example of a hardware functional unit, processing unit 502 may include at least one computing device, such as a server. Alternatively, processing unit 502 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.
[0107] The processing unit 502 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the processing unit 502 includes multiple computing devices that can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the processing unit 502 includes multiple computing devices that can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0108] It should be noted that, in other embodiments, the processing unit 502 can be used to execute any step in the method provided in any of the above embodiments, and the receiving unit 501 can be used to execute any receiving step in the method provided in any of the above embodiments. The steps implemented by the processing unit 502 and the receiving unit 501 can be specified as needed. By implementing different steps in the method provided in any of the above embodiments through the processing unit 502 and the receiving unit 501, all functions of the data processing device 500 can be realized.
[0109] In this embodiment, since the data conversion request received by the receiving unit includes a first quality monitoring rule, the processing unit reads multiple first data stored in the first data table based on the identification information of the first data table, determines the first data that conforms to the first quality monitoring rule, and converts the first data that conforms to the first quality monitoring rule into at least one second data. By converting the first data that conforms to the first quality monitoring rule, the processing unit can exclude first data with quality problems from the first data table and convert the first data that does not have quality problems. Then, based on the identification information of the second data table, the processing unit saves the at least one second data into the second data table, thereby preventing the first data with quality problems in the first data table from being transferred to the second data table during the conversion.
[0110] See Figure 6 This application provides a computing device 600. For example... Figure 6 As shown, the computing device 600 can be Figure 1 or Figure 2 The database system in the network architecture 100 shown, or, could be Figure 3 The database system of method 300 shown. The computing device 600 includes: a bus 602, a processor 604, a memory 606, and a communication interface 608. The processor 604, memory 606, and communication interface 608 communicate via the bus 602. The computing device 600 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 600.
[0111] Bus 602 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 602 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 602 may include a path for transmitting information between various components of the computing device 600 (e.g., memory 606, processor 604, communication interface 608).
[0112] Processor 604 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0113] Memory 606 may include volatile memory, such as random access memory (RAM). Processor 604 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0114] See Figure 6 The memory 606 stores executable program code, and the processor 604 executes this executable program code to implement the functions of the receiving unit 501 and the processing unit 502 in the device 500 shown in the figure, thereby implementing the method provided in any of the above embodiments. That is, the memory 606 stores instructions for executing the method provided in any of the above embodiments. Or,
[0115] The communication interface 603 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 600 and other devices or communication networks.
[0116] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0117] like Figure 7 As shown, the computing device cluster includes at least one computing device 600. The memory 606 of one or more computing devices 600 in the computing device cluster may store the same instructions for performing the methods provided in any of the above embodiments.
[0118] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for performing the data management method described above. In other words, a combination of one or more computing devices 600 can jointly execute instructions for performing the method provided in any of the above embodiments.
[0119] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 8 One possible implementation is shown. For example... Figure 8 As shown, the two computing devices 600A and 600B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.
[0120] In this type of possible implementation, the memory 606 in the computing device 600A stores the execution of, for example Figure 5 Instructions for the function of the processing unit 502 in the illustrated embodiment. Meanwhile, the memory 606 in the computing device 600B stores instructions for executing such... Figure 5 Instructions for the function of the receiving unit 501 in the illustrated embodiment.
[0121] It should be understood that Figure 8 The functions of computing device 600A shown can also be performed by multiple computing devices 600. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600.
[0122] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 The connection method of the computing device cluster. The difference is that the memory 606 of one or more computing devices 600 in the computing device cluster can store the same instructions for executing the methods provided in any of the above embodiments.
[0123] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the methods provided in any of the above embodiments. In other words, a combination of one or more computing devices 600 can jointly execute instructions for performing the methods provided in any of the above embodiments.
[0124] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the methods provided in any of the above embodiments.
[0125] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the method provided in any of the above embodiments.
[0126] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0127] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.
Claims
1. A method of processing data, characterized by, The method includes: A data conversion request is received. The data conversion request includes a first data read statement, a second data read statement, a third data read statement, and a data write statement. The third data read statement includes the identification information of a first data table. The first data read statement includes the identification information of a first column in the first data table and a first quality monitoring rule. The second data read statement includes the identification information of a second column in the first data table and a first quality monitoring rule. The data write statement includes the identification information of a second data table. Based on the identification information of the first data table, read multiple first data stored in the first data table; The first data that conforms to the first quality monitoring rule among the plurality of first data is determined. The first quality monitoring rule is used to monitor each first data for one or more of the following: completeness, validity, consistency, timeliness, or accuracy. Convert the first data that conforms to the first quality monitoring rule into at least one second data; Based on the identification information of the second data table, at least one piece of second data is saved into the second data table; Specifically, the step of reading multiple sets of first data stored in the first data table based on the identification information of the first data table, and determining the first data among the multiple sets of first data that conforms to the first quality monitoring rule, includes: Based on the first data read statement and the identification information of the first data table, the first data in the first column of the first data table is read, and the first data that conforms to the first quality monitoring rule in the first data read statement is determined from the first data in the first column; and based on the second data read statement and the identification information of the first data table, the first data in the second column of the first data table is read, and the first data that conforms to the first quality monitoring rule in the second data read statement is determined from the first data in the second column.
2. The method of claim 1, wherein, The data transformation request also includes a second quality monitoring rule, and the method further includes: Identify the second data among the at least one second data that conforms to the second quality monitoring rule; Saving at least one piece of second data to the second data table based on the identification information of the second data table includes: Based on the identification information of the second data table, the second data that conforms to the second quality monitoring rules is saved into the second data table.
3. The method of claim 2, wherein, The data conversion request includes a data read statement and a data write statement. The data read statement includes the first quality monitoring rule, and the data write statement includes the second quality monitoring rule.
4. The method according to any one of claims 1 to 3, characterized in that, The data conversion request further includes first indication information, which indicates the position of the first quality monitoring rule in the data conversion request. The method further includes: Based on the location indicated by the first indication information, the first quality monitoring rule is read from the data conversion request.
5. An apparatus for processing data, characterized by The device includes: A receiving unit is configured to receive a data conversion request, the data conversion request including a first data read statement, a second data read statement, a third data read statement, and a data write statement. The third data read statement includes identification information of a first data table. The first data read statement includes identification information of a first column in the first data table and a first quality monitoring rule. The second data read statement includes identification information of a second column in the first data table and a first quality monitoring rule. The data write statement includes identification information of a second data table. The processing unit is configured to read multiple first data stored in the first data table based on the identification information of the first data table; The processing unit is further configured to determine the first data among the plurality of first data that conforms to the first quality monitoring rule, wherein the first quality monitoring rule is used to monitor each first data for one or more of the following: completeness, validity, consistency, timeliness or accuracy; The processing unit is further configured to convert the first data that conforms to the first quality monitoring rule into at least one second data. The processing unit is further configured to save the at least one second data into the second data table based on the identification information of the second data table; The processing unit reads multiple sets of first data stored in the first data table based on the identification information of the first data table; and determines the first data among the multiple sets of first data that conforms to the first quality monitoring rule, including: Based on the first data read statement and the identification information of the first data table, the first data in the first column of the first data table is read, and the first data that conforms to the first quality monitoring rule in the first data read statement is determined from the first data in the first column; and based on the second data read statement and the identification information of the first data table, the first data in the second column of the first data table is read, and the first data that conforms to the first quality monitoring rule in the second data read statement is determined from the first data in the second column.
6. The apparatus of claim 5, wherein, The data transformation request also includes a second quality monitoring rule, and the processing unit is used for: Identify the second data among the at least one second data that conforms to the second quality monitoring rule; Based on the identification information of the second data table, the second data that conforms to the second quality monitoring rules is saved into the second data table.
7. The apparatus of claim 6, wherein, The data conversion request includes a data read statement and a data write statement. The data read statement includes the first quality monitoring rule, and the data write statement includes the second quality monitoring rule.
8. The device of any one of claims 5-7, wherein, The data conversion request also includes first indication information, which indicates the position of the first quality monitoring rule in the data conversion request. The processing unit is further configured to read the first quality monitoring rule from the data conversion request based on the position indicated by the first indication information.
9. An apparatus for processing data, the apparatus comprising: It includes at least one processor, the at least one processor being coupled to a memory, reading and executing instructions in the memory to implement the method as claimed in any one of claims 1-4.
10. A computer storage medium having stored thereon a computer program, characterized in that The computer program, which when executed by a processor, implements the method according to any one of claims 1-4.
11. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by a processor, implements the method according to any one of claims 1-4.