Data processing method, data query method and device

Abstract

The application provides a data processing method and device and a data query method, which can be applied to the technical field of databases. The data processing method comprises the following steps: in response to receiving a cross-table query request for a plurality of data tables, performing block processing on the plurality of data tables respectively according to a preset block size to obtain a plurality of block results; combining a plurality of data blocks belonging to different data tables to determine a plurality of intermediate blocks to be processed, wherein each intermediate block to be processed comprises a plurality of to-be-processed row records, and each to-be-processed row record comprises a reference row record in one data block and spliced row records in other data blocks; verifying the plurality of to-be-processed row records in the intermediate blocks to be processed by using the content in the associated columns in which the reference row records and the spliced row records belonging to the same row exist to obtain target row records, so as to generate associated data tables based on the target row records.

Description

Technical Field

[0001] This application relates to the field of database technology, and more specifically to a data processing method, a data query method, and an apparatus. Background Technology

[0002] In big data applications, cross-table queries are a common data retrieval requirement, especially when there are related columns between two tables. Cross-table joins are needed to obtain the complete related data before further retrieval. Traditional cross-table queries typically involve a full table join followed by further filtering based on search criteria.

[0003] In the process of implementing this application, it was found that the relevant technology has at least the following problems: for large-scale data tables, full table joins will consume a lot of memory resources, resulting in slow processing speed and degraded system performance. In addition, the amount of data generated by the full table join in the intermediate process is too large. When the hardware memory space cannot meet the data volume requirements of the full table join result, it is difficult to execute cross-table queries smoothly. Summary of the Invention

[0004] In view of the above problems, this application provides a data processing method, a data query method, and an apparatus.

[0005] According to a first aspect of this application, a data processing method is provided, comprising: responding to receiving a cross-table query request for multiple data tables, dividing the multiple data tables into blocks according to a preset block size to obtain multiple block results, wherein the multiple data tables have the same association column, and the block result includes at least one data block; combining multiple data blocks belonging to different data tables to determine intermediate blocks to be processed, obtaining multiple intermediate blocks to be processed, each intermediate block to be processed including multiple row records to be processed, each row record to be processed including a base row record in one of the data blocks and a concatenated row record in other data blocks; and verifying the multiple row records to be processed in the intermediate blocks to be processed using the contents of the association column in which both the base row record and the concatenated row record of the same row exist, to obtain a target row record, so as to generate an association data table based on the target row record.

[0006] According to an embodiment of this application, the verification of multiple pending row records in the intermediate block to obtain the target row record is performed using the contents of the associated column where both the baseline row record and the concatenated row record belong to the same row. This includes: generating a judgment bitmap based on the associated column and the multiple pending row records, wherein the judgment bitmap includes multiple markers, each marker corresponding to a pending row record, and the markers are used to indicate the processing strategy for the pending row records, the processing strategy including retaining or deleting the pending row records; and processing the multiple pending row records based on the judgment bitmap to obtain the target row record.

[0007] According to an embodiment of this application, a judgment bitmap is generated based on an associated column and multiple pending row records, including: for each pending row record, determining a first element value and a second element value related to the associated column in the pending row record, wherein the first element value is an element value related to the associated column in the base row record and the second element value is an element value related to the associated column in the concatenated row record; and determining a marker corresponding to the pending row record based on the first element value and the second element value.

[0008] According to an embodiment of this application, determining a marker corresponding to a row record to be processed based on a first element value and a second element value includes: if the first element value and the second element value are the same, determining a marker corresponding to the row record to be processed indicates that the row record to be processed is retained; and if the first element value and the second element value are different, determining a marker corresponding to the row record to be processed indicates that the row record to be processed is deleted.

[0009] According to an embodiment of this application, a target row record is obtained by verifying multiple unprocessed row records based on a judgment bitmap, including: determining multiple target markers from the markers included in the judgment bitmap, wherein the target markers are used to indicate that the unprocessed row record corresponding to the target marker is retained; and determining the unprocessed row record corresponding to the target marker as the target row record.

[0010] According to an embodiment of this application, the associated data table is generated in the following manner: for each intermediate block to be processed, multiple rows to be processed within the intermediate block are stored in a storage unit used to cache the intermediate block; the markers in the judgment bitmap are traversed, and the rows to be processed corresponding to the target marker are retained as the target rows, while other rows to be processed that are different from the target rows are removed from the storage unit; the target rows on multiple storage units are summarized to obtain the associated data table.

[0011] According to an embodiment of this application, multiple data blocks belonging to different data tables are combined to determine intermediate blocks to be processed, resulting in multiple intermediate blocks to be processed. This includes: performing a Cartesian product operation on multiple data blocks belonging to different data tables to obtain multiple intermediate blocks to be processed; wherein, when performing the Cartesian product operation, the row records in the data block with a large number of row records are used as the base row records, and the row records in the block with a small number of row records are used as the concatenated row records.

[0012] A second aspect of this application provides a data query method, comprising: in response to receiving a cross-table query request for multiple data tables, determining a related data table of the multiple data tables, wherein the related data table is determined according to the above data processing method; and querying target row records in the related data table according to the query conditions carried in the cross-table query request to obtain query results.

[0013] A third aspect of this application provides a data processing apparatus, comprising: a data table segmentation module, configured to, in response to receiving a cross-table query request for multiple data tables, segment the multiple data tables according to a preset segmentation size to obtain multiple segmentation results, wherein the multiple data tables have the same association column, and the segmentation result includes at least one data block; a data combination module, configured to combine multiple data blocks belonging to different data tables to determine intermediate blocks to be processed, obtaining multiple intermediate blocks to be processed, each intermediate block to be processed including multiple row records to be processed, each row record to be processed including a base row record from one data block and a concatenated row record from other data blocks; and a data verification module, configured to verify the multiple row records to be processed in the intermediate blocks to be processed using the contents of the association column where both the base row record and the concatenated row record belong to the same row, to obtain a target row record, so as to generate an association data table based on the target row record.

[0014] A fourth aspect of this application provides a data query apparatus, comprising: a data association module, configured to determine a related data table of the multiple data tables in response to receiving a cross-table query request for multiple data tables, wherein the related data table is determined according to the above data processing method; and a data query module, configured to query target row records in the related data table according to the query conditions carried in the cross-table query request, and obtain query results.

[0015] A fifth aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0016] A sixth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0017] A seventh aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0018] According to embodiments of this application, by dividing multiple data tables into blocks and combining them to obtain intermediate blocks to be processed, and verifying the intermediate blocks to be processed through the association columns, the target row records are determined, and thus the associated data tables are determined. This refines the table-level data association operations of multiple data tables into association operations at the level of intermediate blocks to be processed, reducing the granularity of operations. This allows the association operations of multiple intermediate blocks to be processed to be executed in parallel, while improving the efficiency and parallelism of data association. Compared with the traditional method of filtering after a full table join, block processing reduces the memory consumption of a single data processing operation and avoids the performance bottleneck that exists when performing a full table join. By using block-by-block concatenation and association column verification, row records that meet the association conditions can be accurately filtered out, improving the efficiency and accuracy of cross-table associations, and making the data processing method adaptable to cross-table query scenarios of large-scale data tables. Attached Figure Description

[0019] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0020] Figure 1 The diagram illustrates an application scenario of the data processing method, data query method, and apparatus according to embodiments of this application.

[0021] Figure 2 A flowchart of a data processing method according to an embodiment of this application is shown.

[0022] Figure 3A A schematic diagram is shown illustrating the determination of a judgment bitmap based on the row record to be processed according to an embodiment of this application.

[0023] Figure 3B This diagram illustrates the filtering of records to be processed based on a judgment bitmap according to an embodiment of this application.

[0024] Figure 4 This paper illustrates a data flow diagram of processing an intermediate block to be processed to obtain an associated data table according to an embodiment of this application.

[0025] Figure 5 A flowchart of a data query method according to an embodiment of this application is shown.

[0026] Figure 6 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown.

[0027] Figure 7 A structural block diagram of a data query apparatus according to an embodiment of this application is shown.

[0028] Figure 8 A block diagram of an electronic device suitable for implementing a data processing method or a data query method according to an embodiment of this application is shown. Detailed Implementation

[0029] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0032] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0033] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, application, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0034] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0035] Embodiments of this application provide a data processing method, comprising: responding to receiving a cross-table query request for a first data table and a second data table, dividing the first data table and the second data table into blocks according to a preset block size to obtain a first block result and a second block result, wherein the first data table and the second data table have the same related columns, the first block result includes at least one first block, and the second block result includes at least one second block; for each first block, combining the first block with each second block to determine at least one intermediate block to be processed, the number of intermediate blocks to be processed being the same as the number of second blocks, the intermediate block to be processed including a left table and a right table, the left table being determined based on the number of row records in the first block and the second block; for each intermediate block to be processed, concatenating the row records in the left table with each row record in the right table to obtain multiple row records to be processed; using the contents of the related columns in which the multiple row records to be processed all exist, verifying the multiple row records to be processed in the multiple intermediate blocks to obtain target row records, so as to generate a related data table based on the multiple target row records.

[0036] Figure 1 The diagram illustrates an application scenario of the data processing method, data query method, and apparatus according to embodiments of this application.

[0037] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0038] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0039] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0040] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0041] It should be noted that the data processing method or data query method provided in the embodiments of this application can generally be executed by server 105. Correspondingly, the data processing device or data query device provided in the embodiments of this application can generally be located in server 105. The data processing method or data query method provided in the embodiments of this application can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data processing device or data query device provided in the embodiments of this application can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0042] It should be understood that Figure 1 The number of first terminal devices, second terminal devices, third terminal devices, networks, and servers shown in the diagram is merely illustrative. Depending on implementation needs, any number of first terminal devices, second terminal devices, third terminal devices, networks, and servers can be included.

[0043] Figure 2 A flowchart of a data processing method according to an embodiment of this application is shown.

[0044] like Figure 2As shown, the data processing method of this embodiment includes operations S210 to S230.

[0045] In operation S210, in response to receiving a cross-table query request for multiple data tables, the multiple data tables are divided into blocks according to a preset block size to obtain multiple block results.

[0046] In operation S220, multiple data blocks belonging to different data tables are combined to determine the intermediate blocks to be processed, resulting in multiple intermediate blocks to be processed.

[0047] In operation S230, the contents of the associated column where both the base row record and the concatenated row record of the same row exist are used to verify multiple unprocessed row records in the intermediate block to obtain the target row record, so as to generate an associated data table based on the target row record.

[0048] If a user's query request for a data table includes multiple data tables in the clause representing the data source, then the query request is a cross-table query request.

[0049] Taking Structured Query Language (SQL) as an example, the clause representing the data source in an SQL statement can be a FROM clause. For example, the SQL statement representing a query request is SELECT * FROM t1, t2 WHERE t1.id = t2.id. The FROM clause is FROM t1, t2, which includes both tables t1 and t2. Therefore, this SQL statement indicates that a query is performed on both tables t1 and t2 to obtain data that satisfies t1.id = t2.id.

[0050] Therefore, the multiple tables targeted by a cross-table query request can be determined based on the clause representing the data source in the cross-table query request.

[0051] Multiple data tables are divided into blocks according to a preset block size, resulting in multiple block results. Each block result includes at least one data block, and the preset block size is used to limit the amount of data included in each data block.

[0052] Taking data table t1 as an example, in the block division result of t1, except for the last data block, the data volume of the other data blocks is the same as the preset block size, and the data volume of the last data block is less than or equal to the preset block size.

[0053] For example, data table t1 contains 45,000 rows of data. The preset block size is 4,096, meaning that each data block contains a maximum of 4,096 rows of data. By dividing the first data table into blocks according to the preset block size, the resulting block structure can include eleven data blocks. The first ten data blocks each contain 4,096 rows of data, and the eleventh data block contains the remaining 4,040 rows of data.

[0054] After the block processing is completed, multiple data blocks belonging to different data tables can be combined. In other words, for each data block obtained after the block processing of data table t1, the data block is combined with each data block obtained after the block processing of data table t2 to obtain the intermediate block to be processed.

[0055] It should be noted that the multiple data tables involved in the aforementioned cross-table query request share the same related columns. After the intermediate blocks to be processed are divided, for each intermediate block, the Cartesian product between the rows of the data blocks that need to be combined can be directly calculated. Within this intermediate block, the base table and the concatenation table for the Cartesian product operation can be determined based on the number of row records in data blocks t1 and t2. Each intermediate block includes multiple row records to be processed, and each row record includes the base row record from one data block and the concatenation row records from other data blocks.

[0056] Specifically, row records in the baseline table of the intermediate block to be processed can be concatenated with each row record in the concatenation table to obtain multiple row records to be processed. It can be understood that the baseline table in this embodiment is the left table in the Cartesian product operation, and the concatenation table is the right table in the Cartesian product operation.

[0057] The above operations will result in a large number of unprocessed row records obtained by directly concatenating the row data in the base table and the row data in the concatenation table. This will lead to a large amount of dirty data in the unprocessed row records that does not conform to the business logic or the nature of the data relationship.

[0058] For example, if both data tables t1 and t2 have an "id" column, then the related column is the "id" column. For a row in the base table, the value corresponding to the related column is 1. For a row in the concatenated table, the value corresponding to the related column is 3. The resulting row to be processed, obtained through concatenation, represents rows where the values ​​corresponding to the related column are both 1 and 3. However, in real-world scenarios, there is no situation where "id" is simultaneously 1 and 3. Therefore, this row to be processed is actually meaningless dirty data.

[0059] Therefore, multiple pending row records in multiple pending intermediate blocks can be traversed. For each pending row record, the data related to the associated column is verified. If the pending row record passes the verification, it is determined as the target row record. If the pending row record fails the verification, it is discarded.

[0060] After traversing all pending row records, a related data table can be generated based on the currently determined target row record, so that further data queries can be performed based on cross-table query requests and the related data table.

[0061] According to embodiments of this application, by dividing multiple data tables into blocks and combining them to obtain intermediate blocks to be processed, and verifying the intermediate blocks to be processed through the association columns, the target row records are determined, and thus the associated data tables are determined. This refines the table-level data association operations of multiple data tables into association operations at the level of intermediate blocks to be processed, reducing the granularity of operations. This allows the association operations of multiple intermediate blocks to be processed to be executed in parallel, while improving the efficiency and parallelism of data association. Compared with the traditional method of filtering after a full table join, block processing reduces the memory usage of a single data processing operation and avoids the performance bottleneck that exists when performing a full table join. By using block-by-block concatenation and association column verification, row records that meet the association conditions can be accurately filtered out, improving the efficiency and accuracy of cross-table associations, and making the data processing method adaptable to cross-table query scenarios of large-scale data tables.

[0062] According to an embodiment of this application, the content of an associated column containing both a baseline row record and a concatenated row record belonging to the same row is used to verify multiple unprocessed row records in the intermediate block to obtain a target row record. This includes: generating a judgment bitmap based on the associated column and multiple unprocessed row records, the judgment bitmap including multiple markers, each marker corresponding to an unprocessed row record, used to represent the processing strategy for the unprocessed row record, the processing strategy including retaining or deleting the unprocessed row record; and processing the multiple unprocessed row records based on the judgment bitmap to obtain the target row record.

[0063] Based on the data related to the associated columns in multiple pending records, it is possible to determine whether multiple pending records conform to business logic and the nature of data association, and accordingly determine the marker in the judgment bitmap corresponding to the pending row record.

[0064] Based on the markers in the bitmap that correspond to the multiple records to be processed, the processing strategy for each record to be processed can be determined, and the corresponding record to be processed can be processed based on the processing strategy for each record to be processed, so as to obtain the target row record to be retained.

[0065] According to embodiments of this application, a method of first generating a judgment bitmap and then processing data based on the judgment bitmap achieves decoupling of judgment and processing. By marking the processing strategy of the row records to be processed using the judgment bitmap, rapid filtering of batch row records is achieved. Compared to the method of processing row by row after comparison, the judgment bitmap can record the processing strategy of all row records to be processed at once, and subsequent batch processing only needs to be performed based on the bitmap markings. This allows for centralized processing of similar operations, thereby improving the efficiency of the verification and filtering process.

[0066] According to an embodiment of this application, a judgment bitmap is generated based on an associated column and multiple pending row records, including: for each pending row record, determining a first element value and a second element value related to the associated column in the pending row record, wherein the first element value is an element value related to the associated column in the base row record and the second element value is an element value related to the associated column in the concatenated row record; and determining a marker corresponding to the pending row record based on the first element value and the second element value.

[0067] Figure 3A A schematic diagram is shown illustrating the determination of a judgment bitmap based on the row record to be processed according to an embodiment of this application.

[0068] like Figure 3A As shown, there are six pending row records, and the row record to be verified is the second row record. For the row record to be verified, determine the base row record and the concatenated row record to obtain the row record. In this example, the base row record includes two elements: 102 corresponding to the id and Zhang San corresponding to the name. The concatenated row record includes two elements: 105 corresponding to the id, Physics corresponding to the subject, and 93 corresponding to the score.

[0069] Since the associated column is id, the element value related to the associated column is determined from the base row record as the first element value, which is 102 in this example, and the element value related to the associated column is determined from the concatenated row record as the second element value, which is 105 in this example.

[0070] Specifically, the element values ​​related to the associated columns can be determined based on the base row record and the header of the base table, and the element values ​​related to the associated columns can be determined based on the concatenated row record and the header of the concatenated table.

[0071] Based on the first and second element values, the authenticity of the row record to be processed and whether it conforms to business logic can be determined. Specifically, based on the first and second element values, a flag corresponding to the row record to be processed is determined. This can include: if the first and second element values ​​are the same, determining the flag corresponding to the row record to be processed indicates that the row record to be processed is retained; and if the first and second element values ​​are different, determining the flag corresponding to the row record to be processed indicates that the row record to be processed is deleted. In this example, if the first and second element values ​​are the same, the flag corresponding to the row record to be processed is 1, indicating that the row record to be processed is retained; if the first and second element values ​​are different, the flag corresponding to the row record to be processed is 0, indicating that the row record to be processed is not retained.

[0072] Understandable, Figure 3A The process of verifying one pending row record is only shown in the example. Other pending row records can also be verified in the same way to obtain the corresponding tags and the judgment bitmap.

[0073] According to the embodiments of this application, the authenticity of the row record to be processed and whether it conforms to the business logic are determined based on the element values ​​corresponding to the associated column in the row record to be processed, thereby verifying and judging the row record to be processed, and providing a clear basis for judgment in the generation of the judgment bitmap.

[0074] According to an embodiment of this application, a target row record is obtained by verifying multiple unprocessed row records based on a judgment bitmap, including: determining multiple target markers from the markers included in the judgment bitmap, wherein the target markers are used to indicate that the unprocessed row record corresponding to the target marker is retained; and determining the unprocessed row record corresponding to the target marker as the target row record.

[0075] Figure 3B This diagram illustrates the filtering of records to be processed based on a judgment bitmap according to an embodiment of this application.

[0076] like Figure 3B As shown, we can first identify multiple target markers in the judgment bitmap, that is, the last three markers in the judgment bitmap are target markers. Combining the correspondence between multiple markers in the judgment bitmap and multiple unprocessed row records, we can identify the unprocessed row records that correspond to the multiple target markers respectively. That is, the last three unprocessed row records correspond to the target markers. We can then identify the last three unprocessed row records as target row records and further determine the associated data table based on the target row records.

[0077] It should be noted that you can first remove duplicates from the elements corresponding to the associated columns in each target row record, then determine the column order of the associated data table based on the column names included in the first and second data tables, and finally write all the target row records into the associated data table according to the column order.

[0078] exist Figure 3B Since both the base table and the concatenation table have an "id" column, there are two id columns in the target row records. After the bitmap filtering, the element values ​​corresponding to the two id columns of the multiple target row records are the same. Therefore, one id column can be deleted by deduplication.

[0079] According to embodiments of this application, target row records are quickly located by filtering target markers and matching corresponding rows to be processed. Valid target row records can be determined directly based on the target markers in the bitmap, eliminating the need to traverse and compare related columns again, thus simplifying the filtering process and further improving data processing efficiency.

[0080] Figure 4 This paper illustrates a data flow diagram of processing an intermediate block to be processed to obtain an associated data table according to an embodiment of this application.

[0081] like Figure 4 As shown, a storage unit can be allocated to cache each intermediate block to be processed. If the number of intermediate blocks to be processed exceeds the number of storage units, the intermediate blocks to be processed are divided into multiple batches according to the number of storage units. The figure shows 4 storage units and 7 intermediate blocks to be processed, so the intermediate blocks to be processed are divided into two batches. The first batch includes 4 intermediate blocks to be processed, and the second batch includes the remaining 3 intermediate blocks to be processed.

[0082] For each intermediate block to be processed, multiple unprocessed row records within the intermediate block are stored in a storage unit used to cache the intermediate blocks to be processed. The markers in the judgment bitmap are traversed, and the unprocessed row records corresponding to the target marker are retained as the target row records. Other unprocessed row records that are different from the target row records are identified. Since the above-mentioned other unprocessed row records are dirty data that do not conform to the business logic, they can be directly removed from the storage unit.

[0083] Taking the intermediate block 1 to be processed as an example, the intermediate block 1 to be processed includes row record 1 to row record 6 to be processed. The six markers in the corresponding judgment bitmap are (0, 1, 0, 0, 1, 1) in sequence. Based on the correspondence between the markers and the row records to be processed, the target row records of the intermediate block 1 to be processed can be determined to be row record 2, row record 5 and row record 6 to be processed.

[0084] By processing other intermediate blocks in the same way, the target row records of each intermediate block can be obtained.

[0085] After processing is completed on multiple storage units, the target row records on these units can be aggregated to obtain an associated data table. Specifically, when processing intermediate blocks in batches, the target row records on each batch can be aggregated after processing of multiple storage units in each batch, before proceeding to the next batch, until all intermediate blocks are processed and the associated data table is obtained.

[0086] According to embodiments of this application, a relational data table is generated through block caching, bitmap traversal, data tagging, and data aggregation, achieving efficient management and aggregation of block data. Block caching alleviates the memory pressure caused by caching the entire dataset, while multi-storage unit aggregation enables rapid integration of block results, ensuring the efficiency of relational data table generation and reducing dependence on hardware resources.

[0087] According to an embodiment of this application, multiple data blocks belonging to different data tables are combined to determine intermediate blocks to be processed, resulting in multiple intermediate blocks to be processed. This includes: performing a Cartesian product operation on multiple data blocks belonging to different data tables to obtain multiple intermediate blocks to be processed; wherein, when performing the Cartesian product operation, the row records in the data block with a large number of row records are used as the base row records, and the row records in the block with a small number of row records are used as the concatenated row records.

[0088] During the Cartesian product operation, after reading one data point from the base table, the entire concatenated table needs to be traversed. When the number of rows in the concatenated table is large, the multiple traversals will increase the pressure on data loading and caching.

[0089] In the embodiments of this application, except for the last first block and the last second block, the number of row records in the other first blocks and second blocks is the same. Therefore, for the intermediate blocks to be processed composed of these first blocks and second blocks with the same number of row records, the base table and the splicing table can be arbitrarily set. For the intermediate blocks to be processed composed of first blocks and second blocks with different numbers of row records, the number of row records in the first blocks and second blocks can be determined, and the block with fewer row records can be used as the splicing table for the intermediate blocks to be processed, so as to reduce the cache pressure caused by frequent traversal of the splicing table.

[0090] According to embodiments of this application, the base table and concatenation table for intermediate blocks to be processed are determined based on the number of row records in the blocks, thereby achieving optimized scheduling of block joins. Using blocks with fewer row records as the concatenation table increases the probability that the cache space used to cache the concatenation table meets the data volume requirements of the concatenation table during Cartesian product operations. This reduces resource waste caused by frequent data reads from the cache space, improves cache hit rate, further enhances the efficiency of block processing, and optimizes the performance of cross-table queries.

[0091] Figure 5 A flowchart of a data query method according to an embodiment of this application is shown.

[0092] like Figure 5 As shown, the data query method in this embodiment includes operations S510 to S520.

[0093] In operation S510, in response to receiving a cross-table query request for multiple data tables, the related data tables of the multiple data tables are determined.

[0094] When operating S520, the target row records in the related data table are queried according to the query conditions carried in the cross-table query request, and the query results are obtained.

[0095] The associated data table can be determined according to the data processing method provided in this application.

[0096] When a cross-table query request is an SQL statement, the query conditions it carries can be the options in the SQL statement. Taking the statement "SELECT * FROM t1, t2 WHERE t1.id = t2.id" as an example, the option "*" indicates that all rows and columns of data in the related table are selected. In this case, the related table can be used as the query result.

[0097] According to the embodiments of this application, based on the query conditions carried in the cross-table query request, further querying and filtering of the related data table can ensure that the target row records are accurately filtered, thus guaranteeing the accuracy of the query results.

[0098] Based on the above data processing method, this application also provides a data processing apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0099] Figure 6 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown.

[0100] like Figure 6 As shown, the data processing device 600 of this embodiment includes a data table segmentation module 610, a data combination module 620, and a data verification module 630.

[0101] The data table partitioning module 610 is used to respond to a cross-table query request for multiple data tables, and to partition the multiple data tables into blocks according to a preset block size to obtain multiple partitioned results. The multiple data tables may have the same related columns, and each partitioned result includes at least one data block. In one embodiment, the data table partitioning module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0102] The data combination module 620 is used to combine multiple data blocks belonging to different data tables, determine intermediate blocks to be processed, and obtain multiple intermediate blocks to be processed. Each intermediate block to be processed includes multiple row records to be processed, and each row record to be processed includes a base row record from one data block and concatenated row records from other data blocks. In one embodiment, the data combination module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0103] The data verification module 630 is used to verify multiple unprocessed row records in the intermediate block by utilizing the contents of the associated column where both the base row record and the concatenated row record belong to the same row, thereby obtaining the target row record, so as to generate an associated data table based on the target row record. In one embodiment, the data verification module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0104] According to an embodiment of this application, the data verification module 630 includes a bitmap generation submodule and a data processing submodule.

[0105] The bitmap generation submodule is used to generate a judgment bitmap based on the associated column and multiple rows to be processed. The judgment bitmap includes multiple markers, each marker corresponding to a row to be processed. The markers are used to indicate the processing strategy for the row to be processed, which includes keeping or deleting the row to be processed.

[0106] The data processing submodule is used to process multiple rows of records to be processed based on the judgment bitmap to obtain the target row record.

[0107] According to an embodiment of this application, the bitmap generation submodule includes an element value determination unit and a marker determination unit.

[0108] The element value determination unit is used to determine, for each row record to be processed, the first element value and the second element value related to the associated column in the row record to be processed. The first element value is the element value related to the associated column in the base row record, and the second element value is the element value related to the associated column in the concatenated row record.

[0109] The tag determination unit is used to determine the tag corresponding to the row record to be processed based on the first element value and the second element value.

[0110] According to an embodiment of this application, the marker determination unit includes a first marker determination subunit and a second marker determination subunit.

[0111] The first marker determination subunit is used to determine the marker corresponding to the row record to be processed, indicating that the row record to be processed is retained, when the first element value and the second element value are consistent.

[0112] The second marker determination subunit is used to determine the marker corresponding to the row record to be processed, indicating that the row record to be processed will be deleted, when the first element value and the second element value are inconsistent.

[0113] According to an embodiment of this application, the data processing submodule includes a marker positioning unit and a record determination unit.

[0114] The marker positioning unit is used to determine multiple target markers from the markers included in the judgment bitmap. The target markers are used to indicate the row records to be processed that are to be retained.

[0115] The record determination unit is used to determine the unprocessed row record corresponding to the target mark as the target row record.

[0116] According to an embodiment of this application, the data processing apparatus 600 further includes a record cache module, a bitmap traversal module, and a record summarization module.

[0117] The record caching module is used to store multiple unprocessed rows within each unprocessed intermediate block into a storage unit used for caching the unprocessed intermediate block.

[0118] The bitmap traversal module is used to traverse the markers in the judgment bitmap, retain the unprocessed row record corresponding to the target marker as the target row record, and remove other unprocessed row records that are different from the target row record from the storage unit.

[0119] The record summarization module is used to summarize target row records from multiple storage units to obtain a related data table.

[0120] According to an embodiment of this application, the data combination module 620 includes a data operation submodule.

[0121] The data operation submodule is used to perform Cartesian product operations on multiple data blocks belonging to different data tables to obtain multiple intermediate blocks to be processed. When performing Cartesian product operations, the row records in the data block with more row records are used as the base row records, and the row records in the block with fewer row records are used as the concatenation row records.

[0122] Based on the above data query method, this application also provides a data query device. The following will combine... Figure 7 The device is described in detail.

[0123] Figure 7 A structural block diagram of a data query apparatus according to an embodiment of this application is shown.

[0124] like Figure 7 As shown, the data query device 700 of this embodiment includes a data association module 710 and a data query module 720.

[0125] The data association module 710 is used to determine the associated data table of the multiple data tables in response to receiving a cross-table query request for multiple data tables, wherein the associated data table is determined according to the data processing method described above. In one embodiment, the data association module 710 can be used to perform the operation S510 described above, which will not be repeated here.

[0126] The data query module 720 is used to query the target row records in the related data table according to the query conditions carried in the cross-table query request, and obtain the query results. In one embodiment, the data query module 720 can be used to perform the operation S520 described above, which will not be repeated here.

[0127] According to embodiments of this application, any multiple modules among the data table segmentation module 610, data combination module 620, data verification module 630, data association module 710, and data query module 720 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the data table segmentation module 610, data combination module 620, data verification module 630, data association module 710, and data query module 720 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the data table segmentation module 610, data combination module 620, data verification module 630, data association module 710, and data query module 720 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0128] Figure 8 A block diagram of an electronic device suitable for implementing a data processing method or a data query method according to an embodiment of this application is shown.

[0129] like Figure 8 As shown, an electronic device 800 according to an embodiment of this application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0130] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0131] According to embodiments of this application, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0132] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0133] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0134] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0135] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0136] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0137] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0138] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0141] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A data processing method, characterized by, The method includes: In response to receiving a cross-table query request for multiple data tables, the multiple data tables are divided into blocks according to a preset block size to obtain multiple block results, wherein the multiple data tables have the same related columns, and the block results include at least one data block; Multiple data blocks belonging to different data tables are combined to determine intermediate blocks to be processed, resulting in multiple intermediate blocks to be processed. Each intermediate block to be processed includes multiple row records to be processed, and each row record to be processed includes a base row record from one of the data blocks and a concatenated row record from other data blocks. Based on the associated column and multiple rows to be processed, a judgment bitmap is generated. The judgment bitmap includes multiple markers, each marker corresponding to a row to be processed. The markers are used to indicate the processing strategy for the row to be processed. The processing strategy includes retaining the row to be processed or deleting the row to be processed. Based on the judgment bitmap, multiple rows to be processed are processed to obtain target rows, so as to generate an associated data table based on the target rows.

2. The method of claim 1, wherein, The step of generating a judgment bitmap based on the associated column and multiple rows to be processed includes: For each of the rows to be processed, determine a first element value and a second element value in the row to be processed that are related to the associated column, wherein the first element value is the element value in the base row record that are related to the associated column, and the second element value is the element value in the concatenated row record that are related to the associated column; and Based on the first element value and the second element value, a tag corresponding to the row record to be processed is determined.

3. The method according to claim 2, characterized in that, The step of determining the tag corresponding to the row record to be processed based on the first element value and the second element value includes: If the first element value and the second element value are the same, determine the flag corresponding to the row record to be processed to indicate that the row record to be processed is retained; and If the first element value and the second element value are inconsistent, the flag corresponding to the pending row record is determined to indicate that the pending row record is deleted.

4. The method according to claim 3, characterized in that, The step of verifying multiple rows to be processed based on the judgment bitmap to obtain the target row record includes: From the markers included in the judgment bitmap, multiple target markers are determined, wherein the target markers are used to indicate that the row record to be processed corresponding to the target marker is retained; and The unprocessed row record corresponding to the target mark is determined as the target row record.

5. The method according to claim 4, characterized in that, The associated data table is generated in the following way: For each of the aforementioned intermediate blocks to be processed The multiple unprocessed line records within the unprocessed intermediate block are stored in a storage unit used to cache the unprocessed intermediate block; The markers in the judgment bitmap are traversed, and the unprocessed row records corresponding to the target marker are retained as the target row record. Other unprocessed row records that are different from the target row record are removed from the storage unit. The target row records on multiple storage units are aggregated to obtain the associated data table.

6. The method according to claim 1, characterized in that, The step of combining multiple data blocks belonging to different data tables to determine intermediate blocks to be processed, resulting in multiple intermediate blocks to be processed, includes: Perform a Cartesian product operation on multiple data blocks belonging to different data tables to obtain multiple intermediate blocks to be processed; Specifically, when performing the Cartesian product operation, the row records in the data block with a large number of row records are used as the base row records, and the row records in the block with a small number of row records are used as the concatenated row records.

7. A data query method, characterized in that, The method includes: In response to receiving a cross-table query request for multiple data tables, a related data table is determined, wherein the related data table is determined by the method according to any one of claims 1 to 5; and Based on the query conditions carried in the cross-table query request, the target row records in the associated data table are queried to obtain the query results.

8. A data processing apparatus, characterized in that, The device includes: The data table partitioning module is used to respond to a cross-table query request for multiple data tables, partition the multiple data tables according to a preset partitioning size, and obtain multiple partitioning results. Among them, there are the same related columns among the multiple data tables, and the partitioning results include at least one data block. The data combination module is used to combine multiple data blocks belonging to different data tables, determine intermediate blocks to be processed, and obtain multiple intermediate blocks to be processed. Each intermediate block to be processed includes multiple row records to be processed, and each row record to be processed includes a base row record in one of the data blocks and a concatenated row record in other data blocks. The bitmap generation submodule is used to generate a judgment bitmap based on the associated column and multiple rows to be processed. The judgment bitmap includes multiple markers, each marker corresponding to a row to be processed. The markers are used to indicate the processing strategy for the row to be processed. The processing strategy includes retaining the row to be processed or deleting the row to be processed. The data processing submodule is used to process multiple rows to be processed based on the judgment bitmap to obtain a target row record, so as to generate an associated data table based on the target row record.

9. A data query device, characterized in that, The device includes: A data association module is configured to, in response to receiving a cross-table query request for multiple data tables, determine the associated data table of the multiple data tables, wherein the associated data table is determined by the apparatus according to claim 8; and The data query module is used to query the target row records in the associated data table according to the query conditions carried in the cross-table query request, and obtain the query results.

Images