Table data processing method, apparatus, device, and medium

Abstract

The application discloses a kind of table data processing method, device, equipment and medium, it is related to computer technical field, the method includes: the primary key data of each target table is de-duplicated, to obtain the primary key data after de-duplication;Wherein, the target table is a kind of data collection, storage and indexing mode;Utilize the target point position of the primary key data after de-duplication in target set obtained using preset hash function;Filter the data not matched with the target point position in each target table, obtain filtered data, so as to utilize the filtered data to execute preset table data processing logic.Can reduce the resources required for table data processing in a variety of scenarios. Utilize the target point position of the primary key data after de-duplication in target set obtained using preset hash function, filter the data not matched with the target point position in each target table, so that the data to be processed subsequently is greatly reduced, without limiting the type of target table, so in a variety of scenarios, the required resources are significantly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a table data processing method and device, equipment and medium. BACKGROUND

[0002] In the era of big data, the correlation analysis between large tables has always been a difficult problem. Currently, it is mainly optimized by increasing cluster resources or adjusting business logic, but most data scenarios can only increase cluster resources to solve. As the amount of data increases, the cost of data processing is getting higher and higher, and the cost of resources is also getting larger and larger. In addition, correlation analysis and other processing can also be performed through a big data computing engine, such as spark, by broadcasting a small table to improve the analysis and calculation performance of table correlation, but the data volume of the broadcast table cannot be large, and too large can easily cause system instability, and the broadcast table cannot be the main table. Therefore, when completing the purpose of table correlation analysis and other processing, there are many restrictions, and it is difficult to complete the table processing task in many situations.

[0003] As can be seen from the above, how to reduce the resources required for table data processing in various scenarios is a problem to be solved in the art. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a table data processing method, device, equipment and medium, which can reduce the resources required for table data processing in various scenarios. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a table data processing method, comprising:

[0006] The primary key data of each target table is de-duplicated to obtain de-duplicated primary key data. The target table is a data collection, storage and indexing method.

[0007] A target point of the de-duplicated primary key data in the target set is obtained by using a preset hash function.

[0008] The data in each target table that does not match the target point is filtered to obtain filtered data, so that the preset table data processing logic is executed using the filtered data.

[0009] Optionally, the de-duplication of the primary key data of each target table to obtain the de-duplicated primary key data comprises:

[0010] The primary key data of each target table is read based on a preset condition, and the primary key data is merged to obtain merged primary key data.

[0011] The merged primary key data of each target table is de-duplicated to obtain de-duplicated primary key data.

[0012] Optionally, the obtaining the target point position of the deduplicated primary key data in the target set by using the preset hash function comprises:

[0013] saving the deduplicated primary key data into a preset storage location, so as to obtain the deduplicated primary key data from the preset storage location and obtain the target point position of the deduplicated primary key data in the target set by using the preset hash function; wherein the preset storage location is any one of a preset memory and a preset third-party storage location.

[0014] Optionally, before the obtaining the target point position of the deduplicated primary key data in the target set by using the preset hash function, the method further comprises:

[0015] applying for a target set of a preset size and determining a preset hash function.

[0016] Optionally, the obtaining the target point position of the deduplicated primary key data in the target set by using the preset hash function comprises:

[0017] calculating a hash function value of the deduplicated primary key data by using the preset hash function, performing a modulo operation on the hash function value and a data amount of the target set to obtain a corresponding target remainder, and determining the target remainder as the target point position of the deduplicated primary key data in the target set.

[0018] Optionally, the obtaining the target point position of the deduplicated primary key data in the target set by using the preset hash function comprises:

[0019] determining a current primary key data from the deduplicated primary key data, obtaining the target point position of the current primary key data in the target set by using the preset hash function, and determining the current primary key data as the marked primary key data.

[0020] determining whether there is unmarked primary key data, if there is, determining a next primary key data from the deduplicated primary key data, updating the next primary key data as the current primary key data, and jumping back to the step of obtaining the target point position of the current primary key data in the target set by using the preset hash function until there is no unmarked primary key data.

[0021] Optionally, the executing the preset table data processing logic by using the filtered data comprises:

[0022] performing table data correlation analysis by using the filtered data.

[0023] and / or performing table data calculation analysis by using the filtered data.

[0024] In a second aspect, the present application discloses a table data processing device, comprising:

[0025] a data deduplication module configured to deduplicate the primary key data of each target table to obtain deduplicated primary key data, wherein the target table is a data collection, storage and indexing method;

[0026] a target point acquisition module configured to obtain a target point of the deduplicated primary key data in a target collection by using a preset hash function;

[0027] a table data processing module configured to filter data in each target table that does not match the target point to obtain filtered data, so as to execute a preset table data processing logic by using the filtered data.

[0028] In a third aspect, the present application discloses an electronic device, comprising:

[0029] a memory configured to save a computer program;

[0030] a processor configured to execute the computer program to implement the steps of the table data processing method disclosed above.

[0031] In a fourth aspect, the present application discloses a computer readable storage medium configured to store a computer program; wherein the computer program is executed by a processor to implement the steps of the table data processing method disclosed above.

[0032] The present application has the following beneficial effects: the primary key data of each target table is deduplicated to obtain deduplicated primary key data, wherein the target table is a data collection, storage and indexing method; a target point of the deduplicated primary key data in a target collection is obtained by using a preset hash function; data in each target table that does not match the target point is filtered to obtain filtered data, so as to execute a preset table data processing logic by using the filtered data. The required resources for table data processing can be reduced in various scenarios. As can be seen, the target point of the deduplicated primary key data in a target collection is obtained by using a preset hash function, and data in each target table that does not match the target point is filtered, so that the data to be processed subsequently is greatly reduced, and because the type of the target table is not limited, the table data processing task can be completed while reducing the amount of data to be processed in various scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0034] Figure 1 A table data processing method flow chart is disclosed in the present application;

[0035] Figure 2 A specific primary key data extraction schematic diagram is disclosed in the present application;

[0036] Figure 3 A specific table data processing method flow chart is disclosed in the present application;

[0037] Figure 4 A specific point position determination schematic diagram is disclosed in the present application;

[0038] Figure 5 Another specific table data processing method flow chart is disclosed in the present application;

[0039] Figure 6 A specific data filtering schematic diagram is disclosed in the present application;

[0040] Figure 7 A table data processing device structure schematic diagram is disclosed in the present application;

[0041] Figure 8 An electronic device structure diagram is disclosed in the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0043] In the big data era, the correlation analysis between large tables has always been a difficult problem. At present, it is mainly optimized by increasing cluster resources or adjusting business logic, but most data scenarios can only increase cluster resources to solve. As the amount of data increases, the cost of data processing is getting higher and higher, and the cost of resources is also getting larger and larger. In addition, correlation analysis and other processing can also be performed through a big data computing engine, such as spark, by broadcasting a small table to improve the analysis and calculation performance of table correlation, but the amount of table data that is broadcasted cannot be large, and too large can easily cause system instability, and the broadcasted table cannot be a primary table. Therefore, when completing the purpose of table correlation analysis and other processing, there are many restrictions and requirements, and it is impossible to complete the table processing task in many situations.

[0044] Therefore, the present application correspondingly provides a table data processing scheme, which can reduce the resources required for table data processing in various scenarios.

[0045] Referring to Figure 1 As shown in the drawings, the embodiments of the present application disclose a table data processing method, comprising:

[0046] Step S11: de-duplicate the primary key data of each target table to obtain de-duplicated primary key data; wherein the target table is a data collection, storage and indexing method.

[0047] In the embodiments, the de-duplication of the primary key data of each target table to obtain de-duplicated primary key data comprises: reading the primary key data of each target table based on a preset condition, and merging the primary key data to obtain merged primary key data; de-duplicating the merged primary key data of each target table to obtain de-duplicated primary key data. For example Figure 2 As shown in a specific primary key data extraction schematic diagram, the target tables are table A and table B, the primary key data in table A and table B is read based on a preset condition, and the primary key data is merged to obtain merged primary key data, and the merged primary key data is de-duplicated to reduce redundant data and reduce the resources required for subsequent data processing, wherein the preset condition can be defined by the user as needed in advance, in this way, when reading the primary key data, only the field needs to be selected.

[0048] Step S12: obtaining the target point of the de-duplicated primary key data in the target set by using a preset hash function.

[0049] In the embodiments, the obtaining the target point of the de-duplicated primary key data in the target set by using a preset hash function comprises: saving the de-duplicated primary key data to a preset storage location, so as to obtain the de-duplicated primary key data from the preset storage location, and obtaining the target point of the de-duplicated primary key data in the target set by using a preset hash function; wherein the preset storage location is any one of a preset memory, a preset third-party storage location. After obtaining the de-duplicated primary key data, the de-duplicated primary key data can be saved to the preset memory or the preset third-party storage location, in this way, the de-duplicated primary key data can be obtained from the preset storage location, so as to obtain the target point of the de-duplicated primary key data in the target set by using a preset hash function.

[0050] In this embodiment, before the target point of the deduplicated primary key data in the target set is obtained by using the preset hash function, the method further includes: applying a target set of a preset size, and determining a preset hash function. The data volume of the deduplicated primary key data is determined, for example, 10, and it is predetermined that the size of the target set is twice the data volume of the deduplicated primary key data, so the preset size is 20. In this way, the target set can contain the target point of all the primary key data, and the size of the target set can be smaller than the data volume of the deduplicated primary key data, but when the size of the target set is greater than the data volume of the deduplicated primary key data, the effect is better. The preset hash function is determined, for example, the hash function is a hash function.

[0051] Step S13: filtering the data in each target table that does not match the target point to obtain filtered data, so as to execute a preset table data processing logic by using the filtered data.

[0052] In this embodiment, the execution of the preset table data processing logic by using the filtered data includes: performing table data correlation analysis by using the filtered data; and / or performing table data calculation analysis by using the filtered data. It can be understood that after the filtered data is obtained, the filtered data can be processed accordingly, for example, table data correlation analysis and table data calculation analysis. Therefore, the data volume of the filtered data at this time is much smaller than the data volume of the primary key data, and the data volume required for subsequent table data processing is greatly reduced, so that the processing efficiency can be improved and the resources required for processing data can be reduced.

[0053] The application has the following beneficial effects: the primary key data of each target table is deduplicated to obtain deduplicated primary key data; the target table is a data set, storage and indexing method; the target point of the deduplicated primary key data in the target set is obtained by using a preset hash function; the data in each target table that does not match the target point is filtered to obtain filtered data, so as to execute a preset table data processing logic by using the filtered data. The resources required for table data processing can be reduced in various scenarios. As can be seen, the target point of the deduplicated primary key data in the target set is obtained by using a preset hash function, and the data in each target table that does not match the target point is filtered. In this way, the data required for subsequent processing is greatly reduced, and because the type of the target table is not limited, the table data processing task can be completed while reducing the data volume required for processing in various scenarios.

[0054] Referring to Figure 3 The embodiment of the application discloses a specific table data processing method, which includes:

[0055] Step S21: The primary key data of each target table is deduplicated to obtain deduplicated primary key data; wherein the target table is a data collection, storage and indexing method.

[0056] Step S22: The hash function value of the deduplicated primary key data is calculated by using a preset hash function, and the hash function value is subjected to modulo operation with the data amount of the target set to obtain a corresponding target remainder, and then the target remainder is determined as the target point of the deduplicated primary key data in the target set.

[0057] It can be understood that the same deduplicated primary key data needs to be processed by using multiple different hash functions, for example Figure 4 A specific point determination diagram is shown in FIG. 1. The hash function values of the deduplicated primary key data are calculated by using hash function 1, hash function 2 and hash function 3, and then the hash function values are subjected to modulo operation with the data amount of the target set to obtain target remainders K1, K3 and K6. The K1, K3 and K6 points are found in the target set, and the K1, K3 and K6 points are determined as the target points. It can be understood that the process of determining the target points is the process of constructing the filter, that is, determining how to filter.

[0058] Step S23: The data in each target table that does not match the target point is filtered to obtain filtered data, so that the preset table data processing logic is executed by using the filtered data.

[0059] After the filter is constructed, the data in the target table that does not match the target point is filtered out, reducing the redundant data that does not meet the condition.

[0060] As can be seen, the deduplicated primary key data is filtered by the filter, and the amount of data required for storage and processing is greatly reduced, facilitating subsequent data calculation and analysis, correlation data analysis and other processing, significantly improving the table data processing performance, and the filter construction process is simple and convenient, easy to implement.

[0061] Referring to FIG. 2 Figure 5 The embodiment of the present application discloses another specific table data processing method, which comprises:

[0062] Step S31: The primary key data of each target table is deduplicated to obtain deduplicated primary key data; wherein the target table is a data collection, storage and indexing method.

[0063] Step S32: The current primary key data is determined from the deduplicated primary key data, the target point of the current primary key data in the target set is obtained by using a preset hash function, and the current primary key data is determined as the marked primary key data.

[0064] It can be understood that the de-duplicated primary key data is several, and the target point of the de-duplicated primary key data is obtained in sequence, for example, the de-duplicated primary key data is data 1 and data 2, and the target point of data 1 and data 2 is not obtained at present, data 1 is determined as the current primary key data from data 1 and data 2, and data 1 is determined as the marked primary key data after the target point of data 1 in the target set is obtained by using the preset hash function.

[0065] Step S33: Determine whether there is unmarked primary key data at present, if there is, determine the next primary key data from the de-duplicated primary key data, update the next primary key data as the current primary key data, and jump to the step of obtaining the target point of the current primary key data in the target set by using the preset hash function again until there is no unmarked primary key data at present.

[0066] In the embodiment, it is determined whether there is unmarked primary key data at present, for example, data 2 is not marked, the next primary key data is data 2 from the de-duplicated primary key data, and data 2 is updated as the current primary key data, that is, data 2 is determined as the marked primary key data after the target point of data 2 in the target set is obtained by using the preset hash function, so there is no unmarked primary key data at present.

[0067] Step S34: Filter the data in each target table that does not match the target point to obtain filtered data, so as to execute the preset table data processing logic by using the filtered data.

[0068] For example Figure 6 As shown in a specific data filtering diagram, the primary key data of table A and table B is read respectively, because K1, K3 and K6 are target points, so the data whose target point is not K1, K3 and K6 in the primary key data of table A and table B is filtered out, that is, the data whose target point is K1, K3 and K6 is left, that is, the filtered data is obtained.

[0069] As can be seen, the present application does not need to limit the type of target table, only needs to extract the primary key data of the target table, and uses the filter to filter correspondingly to obtain the filtered data with greatly reduced data amount, in this way, the data amount to be processed is significantly reduced when subsequent other table data processing is performed, so the processing efficiency can be improved and the required computing resources can be reduced.

[0070] Referring to Figure 7 The embodiment of the present application discloses a table data processing device, which comprises:

[0071] The data de-duplication module 11 is configured to de-duplicate the primary key data of each target table to obtain de-duplicated primary key data; wherein the target table is a data set, a storage and indexing mode;

[0072] The target point position obtaining module 12 is configured to obtain a target point position of the deduplicated primary key data in a target set by using a preset hash function.

[0073] The table data processing module 13 is configured to filter data in each target table that does not match the target point position to obtain filtered data, so as to execute a preset table data processing logic by using the filtered data.

[0074] The application has the following beneficial effects: the primary key data of each target table is deduplicated to obtain deduplicated primary key data; a target point position of the deduplicated primary key data in a target set is obtained by using a preset hash function; data in each target table that does not match the target point position is filtered to obtain filtered data, so as to execute a preset table data processing logic by using the filtered data. The required resources for table data processing can be reduced in various scenarios. As can be seen, the target point position of the deduplicated primary key data in a target set is obtained by using a preset hash function, and data in each target table that does not match the target point position is filtered. In this way, the amount of data that needs to be processed subsequently is greatly reduced. Moreover, the type of target table does not need to be limited. Therefore, in various scenarios, the table data processing task can be completed while reducing the amount of data that needs to be processed.

[0075] In some embodiments, the data deduplication module 11 is specifically configured to:

[0076] The primary key data merging unit is configured to read the primary key data of each target table based on a preset condition, and merge the primary key data to obtain merged primary key data.

[0077] The deduplication unit is configured to deduplicate the merged primary key data of each target table to obtain deduplicated primary key data.

[0078] In some embodiments, the target point position obtaining module 12 is specifically configured to:

[0079] The first point position determining unit is configured to save the deduplicated primary key data into a preset storage location, so as to obtain the deduplicated primary key data from the preset storage location, and obtain a target point position of the deduplicated primary key data in a target set by using a preset hash function. The preset storage location is any one of a preset memory and a preset third-party storage location.

[0080] In some embodiments, the table data processing apparatus is specifically configured to:

[0081] The target set application unit is configured to apply for a target set with a preset size.

[0082] The hash function determination unit is configured to determine a preset hash function.

[0083] In some embodiments, the target point position acquisition module 12 is specifically configured to:

[0084] The second point position determination unit is configured to calculate a hash function value of the deduplicated primary key data by using a preset hash function, perform a modulo operation on the hash function value and a data amount of the target set to obtain a corresponding target remainder, and determine the target remainder as a target point position of the deduplicated primary key data in the target set.

[0085] In some embodiments, the target point position acquisition module 12 is specifically configured to:

[0086] The current marking unit is configured to determine a current primary key data from the deduplicated primary key data, obtain a target point position of the current primary key data in the target set by using a preset hash function, and determine the current primary key data as a marked primary key data.

[0087] The jump unit is configured to determine whether there is unmarked primary key data, if there is, determine a next primary key data from the deduplicated primary key data, update the next primary key data as the current primary key data, and jump back to the step of obtaining the target point position of the current primary key data in the target set by using a preset hash function, until there is no unmarked primary key data.

[0088] In some embodiments, the table data processing module 13 is specifically configured to:

[0089] The preset logic execution unit is configured to perform table data correlation analysis by using the filtered data, and / or perform table data calculation analysis by using the filtered data.

[0090] Further, the embodiment of the present application also provides an electronic device. Figure 8 The electronic device 20 shown in the figure is a structural diagram according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.

[0091] Figure 8 The structural diagram of the electronic device provided by the embodiment of the present application. Specifically, it can include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, the computer program is loaded and executed by the processor 21 to realize the following steps:

[0092] The primary key data of each target table is deduplicated to obtain deduplicated primary key data; wherein the target table is a data collection, storage and indexing method;

[0093] The deduplicated primary key data is obtained in the target set by using a preset hash function.

[0094] The data in each target table that does not match the target point is filtered to obtain filtered data, so that the preset table data processing logic is executed using the filtered data.

[0095] In some embodiments, the processor can specifically implement the following steps by executing the computer program stored in the memory:

[0096] The primary key data of each target table is read based on a preset condition, and the primary key data is merged to obtain merged primary key data;

[0097] The merged primary key data of each target table is deduplicated to obtain deduplicated primary key data.

[0098] In some embodiments, the processor can specifically implement the following steps by executing the computer program stored in the memory:

[0099] The deduplicated primary key data is saved to a preset storage location, so that the deduplicated primary key data is obtained from the preset storage location, and a target point of the deduplicated primary key data in a target set is obtained by using a preset hash function; wherein the preset storage location is any one of a preset memory and a preset third-party storage location.

[0100] In some embodiments, the processor can specifically implement the following steps by executing the computer program stored in the memory:

[0101] A target set of a preset size is applied, and a preset hash function is determined.

[0102] In some embodiments, the processor can specifically implement the following steps by executing the computer program stored in the memory:

[0103] The hash function value of the deduplicated primary key data is calculated by using a preset hash function, and the hash function value and the data amount of the target set are subjected to a modulo operation to obtain a corresponding target remainder, and then the target remainder is determined as the target point of the deduplicated primary key data in the target set.

[0104] In some embodiments, the processor can specifically implement the following steps by executing the computer program stored in the memory:

[0105] determining a current primary key data from the deduplicated primary key data, obtaining a target point of the current primary key data in the target set by using a preset hash function, and determining the current primary key data as a marked primary key data;

[0106] determining whether there is unmarked primary key data, if there is, determining a next primary key data from the deduplicated primary key data, updating the next primary key data as the current primary key data, and jumping to the step of obtaining the target point of the current primary key data in the target set by using the preset hash function again until there is no unmarked primary key data.

[0107] In some embodiments, the processor, by executing the computer program stored in the memory, can further include the following steps:

[0108] performing table data correlation analysis by using the filtered data;

[0109] and / or performing table data calculation analysis by using the filtered data.

[0110] In this embodiment, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device; the communication interface 24 is configured to create a data transmission channel between the electronic device and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input and output interface 25 is configured to obtain external input data or output data to the outside world, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0111] The processor 21 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 21 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 21 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0112] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223, etc. The storage mode can be temporary storage or permanent storage.

[0113] The operating system 221 is used to manage and control each hardware device on the electronic device and the computer program 222 to realize the operation and processing of the processor 21 on the mass data 223 in the memory 22. It can be Windows, Unix, Linux, etc. The computer program 222 can further include computer programs for completing other specific work in addition to the computer programs for completing the table data processing method disclosed by the electronic device in any of the preceding embodiments. The data 223 can include data received by the electronic device from external devices, data collected by the input and output interface 25, etc.

[0114] Further, the embodiments of the present application also disclose a computer readable storage medium, the storage medium stores a computer program, and the computer program is loaded and executed by the processor to realize the method steps executed by the table data processing process disclosed in any of the preceding embodiments.

[0115] Finally, it needs to be pointed out that in this article, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0116] The above describes in detail a table data processing method, device, equipment and medium provided by the present application. The principles and implementation manners of the present application are described by applying specific examples in this article. The above example is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for processing table data, characterized in that, include: The primary key data of each target table is deduplicated to obtain the deduplicated primary key data; wherein, the target table is a data collection, storage and indexing method; The target location of the deduplicated primary key data in the target set is obtained using a preset hash function; Data that does not match the target location in each of the target tables is filtered to obtain filtered data, which is then used to execute the preset table data processing logic. The step of deduplicating the primary key data of each target table to obtain deduplicated primary key data includes: The primary key data of each target table is read based on preset conditions, and the primary key data is merged to obtain merged primary key data; the merged primary key data of each target table is deduplicated to obtain deduplicated primary key data. The step of obtaining the target location of the deduplicated primary key data in the target set using a preset hash function includes: The hash function value of the deduplicated primary key data is calculated using a preset hash function, and a modulo operation is performed between the hash function value and the data volume of the target set to obtain the corresponding target remainder. Then, the target remainder is determined as the target position of the deduplicated primary key data in the target set.

2. The table data processing method according to claim 1, characterized in that, The step of obtaining the target location of the deduplicated primary key data in the target set using a preset hash function includes: The deduplicated primary key data is saved to a preset storage location so that the deduplicated primary key data can be retrieved from the preset storage location, and the target position of the deduplicated primary key data in the target set can be obtained using a preset hash function; wherein, the preset storage location is any one of preset memory or preset third-party storage location.

3. The table data processing method according to claim 1, characterized in that, Before obtaining the target location of the deduplicated primary key data in the target set using a preset hash function, the method further includes: Request a target set of a preset size and determine the preset hash function.

4. The table data processing method according to any one of claims 1 to 3, characterized in that, The step of obtaining the target location of the deduplicated primary key data in the target set using a preset hash function includes: The current primary key data is determined from the deduplicated primary key data, and the target position of the current primary key data in the target set is obtained using a preset hash function. The current primary key data is then identified as the marked primary key data. Determine if there is any untagged primary key data. If so, determine the next primary key data from the deduplicated primary key data, update the next primary key data to the current primary key data, and jump back to the step of obtaining the target position of the current primary key data in the target set using a preset hash function, until there is no more untagged primary key data.

5. The table data processing method according to claim 1, characterized in that, The step of using the filtered data to execute the preset table data processing logic includes: Perform table data association analysis using the filtered data; And / or, use the filtered data to perform table data calculation and analysis.

6. A table data processing device, characterized in that, The steps for implementing the table data processing method as described in any one of claims 1 to 5 include: The data deduplication module is used to deduplicate the primary key data of each target table to obtain the deduplicated primary key data; wherein, the target table is a data collection, storage and indexing method; The target location acquisition module is used to obtain the target location of the deduplicated primary key data in the target set using a preset hash function; The table data processing module is used to filter data in each of the target tables that do not match the target location to obtain filtered data, so as to use the filtered data to execute preset table data processing logic.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the table data processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the table data processing method as described in any one of claims 1 to 5.

Images