Data processing method and device, computer device, and storage medium

Abstract

This application relates to a data processing method, apparatus, device, and storage medium, and pertains to the field of big data technology. The method includes: obtaining a data skew key in a data file whose data volume exceeds a data volume threshold; pre-aggregating the values ​​corresponding to the data skew key to obtain pre-aggregated data; the pre-aggregated data contains one data entry; based on the pre-aggregation result corresponding to the data skew key, performing a Map join operation on all values ​​corresponding to the data skew key to obtain a first data table; and performing data processing based on the first data table. This method avoids shuffle operations during data processing, thereby preventing the situation where values ​​with the same data skew key are all sent to the same Reduce function for processing, and thus avoiding data skew during data processing.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the continuous and in-depth development of internet technology applications in today's society, the global data volume has exploded, making data one of the fastest-growing resources. Faced with this rapidly growing, massive, and complex data resource, traditional data analysis and processing technologies have been presented with enormous challenges. Enterprises may experience significant uneven distribution of data processing. Therefore, there is an urgent need for big data technologies and applications to provide solutions that avoid data skew. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and storage medium that can avoid shuffle operations during data processing, thereby preventing the same data skew key value from being sent to the same Reduce function for processing, and thus avoiding data skew during data processing. The technical solution is as follows:

[0004] On the one hand, a data processing method is provided, the method comprising:

[0005] Obtain the data skew key in the data file whose data volume is greater than the data volume threshold;

[0006] By pre-aggregating the values ​​corresponding to the data skew keys, pre-aggregated data corresponding to the data skew keys is obtained; the pre-aggregated data contains one data item.

[0007] Based on the pre-aggregation result corresponding to the data tilt key, a first data table is obtained by performing a Map join operation on all values ​​corresponding to the data tilt key.

[0008] Data processing is performed based on the first data table.

[0009] On the other hand, a data processing apparatus is provided, the apparatus comprising:

[0010] The first acquisition module is used to acquire data skew keys in the data file whose data volume is greater than the data volume threshold.

[0011] The pre-aggregation module is used to pre-aggregate the values ​​corresponding to the data skew key to obtain pre-aggregated data corresponding to the data skew key; the pre-aggregated data contains one data item.

[0012] The first connection module is used to obtain a first data table by performing a Map join operation on all values ​​corresponding to the data tilt key based on the pre-aggregation result corresponding to the data tilt key;

[0013] The data processing module is used to process data based on the first data table.

[0014] In one possible implementation, the pre-aggregation module is used to obtain the pre-aggregated data by concatenating all values ​​corresponding to the same key of the data skew into a string in a target format.

[0015] In one possible implementation, the device further includes:

[0016] The second acquisition module is used to acquire ordinary keys in the data file whose data volume is less than the data volume threshold.

[0017] The second connection module is used to obtain a second data table by performing a join operation on the values ​​corresponding to the ordinary keys;

[0018] The data processing module is used to perform data processing based on the first data table and the second data table.

[0019] In one possible implementation, the data processing module includes:

[0020] The merging submodule is used to obtain a merged data table by merging the first data table and the second data table;

[0021] The partitioning submodule is used to obtain partitioning results by performing partitioning processing based on the merged data table;

[0022] The data processing submodule is used to perform data processing based on the partitioning results.

[0023] In one possible implementation, the merging submodule is used to merge the first data table and the second data table using a Broadcast variable and a Map-type operator to obtain the merged data table.

[0024] In one possible implementation, the merging submodule includes:

[0025] A variable creation unit is used to create the Broadcast variable corresponding to the second data table;

[0026] The processing unit is used to process the first data table and the broadcast variable using the Map operator to obtain the merged data table; the Map operator is used to obtain the full data of the second data table from the broadcast variable, and to join the data with the same key in the first data table and the second data table according to the target method.

[0027] In one possible implementation, the first connection module is used to perform the Map join operation on all values ​​corresponding to the data skew key using a Map-type operator based on the pre-aggregation result corresponding to the data skew key, to obtain the first data table.

[0028] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-described data processing method.

[0029] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer program, which is loaded and executed by a processor to implement the above-described data processing method.

[0030] On the other hand, a computer program product is provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor to implement the data processing methods provided in the various optional implementations described above.

[0031] The technical solution provided in this application may include the following beneficial effects:

[0032] The data processing method provided in this application pre-aggregates the values ​​of data skew keys to obtain pre-aggregated data containing one data entry. After data aggregation, a Mapjoin operation is performed on all values ​​corresponding to the data skew key to generate a first data table. Data processing is then performed based on the first data table. This method avoids shuffle operations during data processing, thereby preventing the situation where values ​​with the same data skew key are all sent to the same Reduce function for processing, and thus avoiding data skew during data processing.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] Figure 1 A flowchart of a data processing method provided in an exemplary embodiment of this application is shown;

[0036] Figure 2 A flowchart of another data processing method provided by an exemplary embodiment of this application is shown;

[0037] Figure 3 A schematic diagram illustrating a data processing procedure provided in an exemplary embodiment of this application is shown;

[0038] Figure 4 A block diagram of a data processing apparatus provided in an exemplary embodiment of this application is shown;

[0039] Figure 5 This is a structural block diagram of a computer device according to an exemplary embodiment;

[0040] Figure 6 This is a structural block diagram of a computer device according to an exemplary embodiment. Detailed Implementation

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0042] Hadoop is a distributed system architecture used to solve the problems of storing and analyzing massive amounts of data. It typically consists of HDFS (Hadoop Distributed File System), YARN (Yet Another Resource Negotiator), and MapReduce. MapReduce divides the computation process into two phases: the Map phase and the Reduce phase. The Map phase processes the input data in parallel, while the Reduce phase aggregates the Map results. Data skew is essentially the uneven distribution of keys, which frequently occurs in MapReduce, usually in the Reduce phase. Because the data is distributed across different Reduce processes, some Reduce tasks are particularly heavy, and the processing time varies between different Reduce processes, resulting in a longer data processing time in the Reduce phase.

[0043] Typically, MapReduce involves a Map process, a Shuffle process, and a Reduce process. The Shuffle process is used to group together the same key and distribute them to the same Reduce function. If the amount of data corresponding to a certain key is large, data skew will occur. Therefore, this application provides a data processing method that avoids data skew by bypassing Shuffle-type operations.

[0044] Figure 1 A flowchart illustrating an exemplary embodiment of this application is shown. This data processing method can be executed by a computer device, which can be implemented as a server or terminal with data processing capabilities, such as... Figure 1 As shown, the data processing method may include the following steps:

[0045] Step 110: Obtain the data skew key in the data file whose data volume is greater than the data volume threshold.

[0046] In this embodiment of the application, the computer device can count the amount of data corresponding to each key in the data file and compare it with the data amount threshold to obtain the data skew key from each key; wherein, the value of the data threshold can be set based on actual needs, and this application does not limit it.

[0047] Step 120: By pre-aggregating the values ​​corresponding to the data skew keys, pre-aggregated data corresponding to the data skew keys is obtained; the pre-aggregated data contains one data item.

[0048] One piece of data in the pre-aggregated data can indicate all the value information corresponding to the data skew key.

[0049] Step 130: Based on the pre-aggregation results corresponding to the data skew key, obtain the first data table by performing a Map join operation on all values ​​corresponding to the data skew key.

[0050] Map join refers to a join operation performed within a Map task, therefore, there is no need to start a Reduce task; Map join is suitable for joining a large table and a small table.

[0051] Step 140: Perform data processing based on the first data table.

[0052] In summary, the data processing method provided in this application pre-aggregates the values ​​of the data skew key to obtain pre-aggregated data containing one data entry. After data aggregation, a Map join operation is performed on all values ​​corresponding to the data skew key to generate a first data table. Data processing is then performed based on the first data table. This method avoids the Shuffle operation during data processing, thereby preventing the situation where values ​​with the same data skew key are all sent to the same Reducer for processing, and thus avoiding data skew during data processing.

[0053] In this embodiment of the application, the data file contains not only data slant keys but also ordinary keys; the amount of data corresponding to the ordinary keys is less than the data amount threshold. Therefore, when processing data, it is necessary to combine data slant keys and ordinary keys for data processing. Figure 2 The flowchart shown is another data processing method provided by an exemplary embodiment of this application. This data processing method can be executed by a computer device, which can be implemented as a server or terminal with data processing capabilities, such as... Figure 2 As shown, the data processing method may include the following steps:

[0054] Step 210: Obtain the data skew key in the data file whose data volume is greater than the data volume threshold.

[0055] The data file can be obtained from a source database based on a pre-defined data collection method; the target database can be a Hive database. The format of the data file can be determined by the data collection method.

[0056] Step 220: By pre-aggregating the values ​​corresponding to the data skew keys, pre-aggregated data corresponding to the data skew keys is obtained; the pre-aggregated data contains one data item.

[0057] Optionally, the values ​​of the data skewed data are pre-aggregated in the Hive table. In this embodiment, the computer device can group the data according to the key and obtain pre-aggregated data by concatenating all the values ​​corresponding to the same data skewed key into a string in the target format. By pre-aggregating the values ​​corresponding to the data skewed key, multiple fields that originally needed to be joined are concatenated into one field in advance, reducing the number of fields to be joined and thus simplifying the complexity of subsequent Spark-side joins.

[0058] The target format can be approved based on actual needs, and this application does not impose any restrictions on it.

[0059] As an illustration, if the target format is concatenated using the symbol "|", during preprocessing, Hive SQL can group and aggregate the products table according to product_id, and connect the name, category, and price fields corresponding to each product_id with "|" to form a product_info string, thereby completing the pre-aggregation of the values ​​corresponding to each product_id.

[0060] In some possible scenarios, the aggregation granularity of the key can also be increased. For example, when performing aggregation, instead of aggregating data by user order ID, data can be aggregated by user ID to increase the aggregation granularity of the key, thereby reducing the amount of data processing required later.

[0061] Step 230: Based on the pre-aggregation results corresponding to the data skew key, obtain the first data table by performing a Map join operation on all values ​​corresponding to the data skew key.

[0062] In this embodiment, the computer device can perform a Map join operation on all values ​​corresponding to the data skew key based on the pre-aggregation result corresponding to the data skew key, using a Map operator to obtain a first data table. By performing a Map join on all values ​​corresponding to the data skew key, other fields in the data skew key can be concatenated into one field, avoiding performing join operations on each field individually.

[0063] Step 240: Obtain the ordinary keys in the data file whose data volume is less than the data volume threshold.

[0064] The process of acquiring data using tilt keys and acquiring data using normal keys can be performed in parallel by the computer device.

[0065] Step 250: Obtain the second data table by performing a Join operation on the values ​​corresponding to the ordinary keys.

[0066] Step 260: Perform data processing based on the first data table and the second data table.

[0067] In this embodiment of the application, the above process can be implemented as follows:

[0068] By merging the first data table and the second data table, a merged data table is obtained;

[0069] The partitioning results are obtained by performing partitioning based on the merged data table;

[0070] Data processing is performed based on the partitioning results.

[0071] The merged data table contains the information required for data processing. For example, if the order to be processed is an order, the fields include (order_id, product_id, quantity...) and the product is a product, the fields include (product_id, name, category, price...). By concatenating the orders and products, detailed product information for each order can be obtained.

[0072] Figure 3 A schematic diagram of a data processing procedure provided in an exemplary embodiment of this application is shown, such as... Figure 3 As shown, after the computer device obtains the data file from the data source, it determines and obtains the data skew key and the normal key in the data file. For the data skew key, the data is first pre-aggregated in the data table to obtain the pre-aggregated data corresponding to the data skew key. Then, a Map join operation is performed on all values ​​of the same data skew key to obtain a first data table. This first data table can be a large RDD (Resilient Distributed Dataset). For the normal key, a join operation is performed on all values ​​of the same normal key to obtain a second data table. This second data table can be a smaller RDD. The first data table and the second data table are merged to obtain a merged data table. Data processing is performed based on partitioning of this merged data table, so that each partitioner can process the data evenly, thereby reducing the impact on the data partitioners and reducing the impact of data skew.

[0073] Specifically, when merging the first data table and the second data table to obtain a merged data table, the computer device can use a Broadcast variable and a Map-type operator to merge the first data table and the second data table to obtain a merged data table.

[0074] Optionally, a Broadcast variable and a Map-type operator can be used to merge the first and second data tables to obtain a merged data table, including:

[0075] Create a Broadcast variable corresponding to the second data table;

[0076] The first data table and the broadcast variable are processed by a Map operator to obtain a merged data table. This Map operator is used to obtain the full data of the second data table from the broadcast variable and join the data with the same key in the first and second data tables according to the target method.

[0077] In this process, when creating the broadcast variable, the computer device identifies which of the two RDDs participating in the Join operation is the "smaller table," i.e., the one with less data. The smaller table RDD is then fetched to the Driver using the `collect` operator, and a broadcast variable is created to broadcast the smaller table data to the memory of each Executor. The Map operator Join executes map-like operators on the larger RDD. In each map task, all data from the smaller table is retrieved from the broadcast variable, and then each element in the larger table RDD is traversed according to the join key to find the corresponding data. The data from the two RDDs are then concatenated as needed to complete the Join. In this embodiment, the computer device can directly fetch the second data table to the Driver's memory using the `collect` operator, and then create a broadcast variable corresponding to the second data table. Next, a Map operator is executed on the first data table. Within the Map operator function, the full data of the second data table is obtained from the Broadcast variable and compared with each data in the first data table according to the key. If the keys are the same, the data corresponding to that key in the two data tables are concatenated.

[0078] The connection method for the same key in the two data tables can be set according to actual needs, and this application does not impose any restrictions on this.

[0079] In summary, the data processing method provided in this application pre-aggregates the values ​​of the data skew key to obtain pre-aggregated data containing one data entry. After data aggregation, a Map join operation is performed on all values ​​corresponding to the data skew key to generate a first data table. Data processing is then performed based on the first data table. This method avoids the Shuffle operation during data processing, thereby preventing the situation where values ​​with the same data skew key are all sent to the same Reducer for processing, and thus avoiding data skew during data processing.

[0080] Figure 4 The diagram illustrates a block diagram of a data processing apparatus provided in an exemplary embodiment of this application, which can be used to perform, for example... Figure 1 or Figure 2 All or part of the steps of the data processing method shown, such as Figure 4 As shown, the data processing apparatus may include:

[0081] The first acquisition module 410 is used to acquire the data skew key in the data file whose data volume is greater than the data volume threshold.

[0082] The pre-aggregation module 420 is used to pre-aggregate the values ​​corresponding to the data skew key to obtain pre-aggregated data corresponding to the data skew key; the pre-aggregated data contains one data item.

[0083] The first connection module 430 is used to obtain a first data table by performing a Map join operation on all values ​​corresponding to the data tilt key based on the pre-aggregation result corresponding to the data tilt key;

[0084] The data processing module 440 is used to perform data processing based on the first data table.

[0085] In one possible implementation, the pre-aggregation module 420 is used to obtain the pre-aggregated data by concatenating all values ​​corresponding to the same key of the data skew into a string in a target format.

[0086] In one possible implementation, the device further includes:

[0087] The second acquisition module is used to acquire ordinary keys in the data file whose data volume is less than the data volume threshold.

[0088] The second connection module is used to obtain a second data table by performing a join operation on the values ​​corresponding to the ordinary keys;

[0089] The data processing module 440 is used to perform data processing based on the first data table and the second data table.

[0090] In one possible implementation, the data processing module 440 includes:

[0091] The merging submodule is used to obtain a merged data table by merging the first data table and the second data table;

[0092] The partitioning submodule is used to obtain partitioning results by performing partitioning processing based on the merged data table;

[0093] The data processing submodule is used to perform data processing based on the partitioning results.

[0094] In one possible implementation, the merging submodule is used to merge the first data table and the second data table using a Broadcast variable and a Map-type operator to obtain the merged data table.

[0095] In one possible implementation, the merging submodule includes:

[0096] A variable creation unit is used to create the Broadcast variable corresponding to the second data table;

[0097] The processing unit is used to process the first data table and the broadcast variable using the Map operator to obtain the merged data table; the Map operator is used to obtain the full data of the second data table from the broadcast variable, and to join the data with the same key in the first data table and the second data table according to the target method.

[0098] In one possible implementation, the first connection module 430 is used to perform the Map join operation on all values ​​corresponding to the data skew key using a Map-type operator based on the pre-aggregation result corresponding to the data skew key, to obtain the first data table.

[0099] In summary, the data processing apparatus provided in this application pre-aggregates the values ​​of the data skew key to obtain pre-aggregated data containing one data entry. After data aggregation, it performs a Map join operation on all values ​​corresponding to the data skew key to generate a first data table, and then performs data processing based on the first data table. This method avoids the Shuffle operation during data processing, thereby preventing the situation where values ​​with the same data skew key are all sent to the same Reducer for processing, and thus avoiding data skew during data processing.

[0100] Figure 5 A structural block diagram of a computer device 500 illustrated in an exemplary embodiment of this application is shown. This computer device can be implemented as a server as described in the above-described scheme of this application. When the computer device is implemented as a server, the server can display a visual configuration interface through an external display device.

[0101] The computer device 500 includes a central processing unit (CPU) 501, a system memory 504 including random access memory (RAM) 502 and read-only memory (ROM) 503, and a system bus 505 connecting the system memory 504 and the CPU 501. The computer device 500 also includes a mass storage device 506 for storing an operating system 509, application programs 510, and other program modules 511.

[0102] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 504 and mass storage device 506 described above can be collectively referred to as memory.

[0103] According to various embodiments of this application, the computer device 500 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 500 can be connected to a network 508 via a network interface unit 507 connected to the system bus 505, or it can use the network interface unit 507 to connect to other types of networks or remote computer systems (not shown).

[0104] The memory further includes at least one instruction, at least one program, code set, or instruction set, which are stored in the memory. The central processing unit 501 executes the at least one instruction, at least one program, code set, or instruction set to implement all or part of the steps in the data processing methods shown in the above embodiments.

[0105] Figure 6 A structural block diagram of a computer device 600 illustrating an exemplary embodiment of this application is shown. The computer device 600 can be implemented as the aforementioned terminal, such as a smartphone, tablet computer, laptop computer, desktop computer, smartwatch, and television. The computer device 600 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0106] Typically, computer device 600 includes a processor 601 and a memory 602.

[0107] In some embodiments, the computer device 600 may optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, and a power supply 608.

[0108] In some embodiments, the computer device 600 further includes one or more sensors 609. The one or more sensors 609 include, but are not limited to, an accelerometer 610, a gyroscope 611, a pressure sensor 612, an optical sensor 613, and a proximity sensor 614.

[0109] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the computer device 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0110] In one exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to implement all or part of the steps in the data processing method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0111] In one exemplary embodiment, a computer program product is also provided, comprising at least one computer program that is loaded and executed by a processor. Figure 1 or Figure 2 All or part of the steps of the data processing method shown in any embodiment.

[0112] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0113] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Retrieve the skew keys in the data file whose data volume exceeds a data volume threshold; By pre-aggregating the values ​​corresponding to the data skew keys, pre-aggregated data corresponding to the data skew keys is obtained; the pre-aggregated data contains one data item. Based on the pre-aggregated data corresponding to the data tilt key, a first data table is obtained by performing a Map join operation on all values ​​corresponding to the data tilt key. Data processing is performed based on the first data table; The step of obtaining pre-aggregated data corresponding to the data skew key by pre-aggregating the values ​​corresponding to the data skew key includes: The pre-aggregated data is obtained by concatenating all values ​​corresponding to the same data slant key into a string in a target format; The method further includes: Obtain ordinary keys in the data file whose data volume is less than the data volume threshold; A second data table is obtained by performing a join operation on the values ​​corresponding to the ordinary keys; The data processing based on the first data table includes: Data processing is performed based on the first data table and the second data table; The data processing based on the first data table and the second data table includes: By merging the first data table and the second data table, a merged data table is obtained; The partitioning results are obtained by performing partitioning processing based on the merged data table; Data processing is performed based on the partitioning results.

2. The method according to claim 1, characterized in that, The step of merging the first data table and the second data table to obtain a merged data table includes: The first data table and the second data table are merged using a Broadcast variable and a Map-type operator to obtain the merged data table.

3. The method according to claim 2, characterized in that, The step of merging the first data table and the second data table using a Broadcast variable and a Map-type operator to obtain the merged data table includes: Create the Broadcast variable corresponding to the second data table; The first data table and the broadcast variable are processed by the Map operator to obtain the merged data table; the Map operator is used to obtain the full data of the second data table from the broadcast variable, and to join the data with the same key in the first data table and the second data table according to the target method.

4. The method according to claim 1, characterized in that, The first data table is obtained by performing a Map join operation on all values ​​corresponding to the data skew key based on the pre-aggregated data, including: Based on the pre-aggregated data corresponding to the data skew key, the Map join operation is performed on all values ​​corresponding to the data skew key using a Map operator to obtain the first data table.

5. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire data skew keys in the data file whose data volume is greater than the data volume threshold. The pre-aggregation module is used to pre-aggregate the values ​​corresponding to the data skew key to obtain pre-aggregated data corresponding to the data skew key; the pre-aggregated data contains one data item. The first connection module is used to obtain a first data table based on the pre-aggregated data corresponding to the data tilt key by performing a Map join operation on all values ​​corresponding to the data tilt key; The data processing module is used to process data based on the first data table; The pre-aggregation module is used to obtain the pre-aggregated data by concatenating all values ​​corresponding to the same data skew key into a string in a target format; The device further includes: The second acquisition module is used to acquire ordinary keys in the data file whose data volume is less than the data volume threshold. The second connection module is used to obtain a second data table by performing a join operation on the values ​​corresponding to the ordinary keys; The data processing module is used to perform data processing based on the first data table and the second data table; The data processing module includes: The merging submodule is used to obtain a merged data table by merging the first data table and the second data table; The partitioning submodule is used to obtain partitioning results by performing partitioning processing based on the merged data table; The data processing submodule is used to perform data processing based on the partitioning results.

6. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to implement the data processing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1 to 4.

Images