Entity resolution parallel processing method, device, equipment and medium

Abstract

The invention is suitable for the technical field of big data mining, and particularly relates to an entity resolution parallel processing method and device, equipment and a medium. According to the method, entity tuples in a to-be-resolved data set are divided into non-intersecting partitions, the entity tuples in any partition are equally divided into N intervals, M continuous intervals form a window, all the windows are subjected to serial processing, a thread block is distributed to each interval in the window for any window, and a thread block is distributed to each interval in the window. The thread blocks are used for matching target tuples in the corresponding intervals with entity tuples except the target tuples in the partitions to obtain matching results, and the thread blocks in the window are processed in parallel; and if not, processing the interval in the next window of the window by using the executed thread block until all the intervals are processed by the thread block to obtain all matching results. The entity data is effectively segmented, resource utilization and parallelization are optimized, and the entity resolution efficiency is improved.

Description

Technical Field

[0001] The present application is applicable to the field of big data mining technology, and in particular relates to a parallel processing method, device, equipment and medium for entity resolution. Background Art

[0002] Entity resolution refers to identifying all data in a dataset that points to the same real entity. Entity resolution methods can be mainly divided into two categories: rule-based methods and deep learning model-based methods. In comparison, rule-based entity resolution methods have the advantages of high efficiency and easy explanation. This method uses a series of rules to filter out mismatches in the dataset and identify potential matching entities. Existing large-scale rule-based entity resolution systems rely on multiple central processing units (CPUs) in a shared-nothing architecture to drive machines. They use large-scale computing clusters to integrate multiple computer resources to complete entity resolution tasks. The specific approach is divided into the following steps: (1) In a shared-nothing architecture, data is partitioned and distributed across multiple processing units. Each computing unit uses local memory to clean data independently; (2) Data that originally belonged to the same entity may be assigned to different processing units. In order to avoid missing these matches, different processing units need to transmit local data to remote processing units through computer networks; (3) After all processing units complete the entity resolution task, the results of entity resolution on each machine are collected.

[0003] However, this approach has certain limitations. For example, (1) these systems are usually expensive to maintain and build, making large-scale graph computing a privilege of a few companies that are capable of maintaining large-scale computer clusters. (2) These distributed computing engines are usually based on the assumption that "the more machines used, the shorter the time it takes for the entire cluster system to complete a computing task." However, this assumption does not always hold true. More computing nodes mean greater communication costs between potential nodes, which means that for some scenarios, no matter how the computing cluster is horizontally expanded, the system performance cannot be significantly improved. (3) In order to avoid missed matches, different processing units need to transmit local data to remote processing units through computer networks, which brings additional data transmission overhead.

[0004] In view of the above problems, new hardware acceleration technologies and parallel processing strategies can be explored to improve the performance of entity resolution. Graphic Processing Unit (GPU), as a processing unit based on shared memory architecture, has the ability of efficient data sharing, collaboration between processing units, and dynamic workload scheduling, providing a parallel processing solution for accelerating rule-based entity resolution. Therefore, how to effectively split entity data to optimize resource utilization and parallelism to improve entity resolution efficiency has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, embodiments of the present application provide a method, apparatus, device and medium for parallel processing of entity resolution to solve the problem of how to effectively split entity data to optimize resource utilization and parallelism to improve entity resolution efficiency.

[0006] In a first aspect, an embodiment of the present application provides a parallel processing method for entity resolution, the parallel processing method for entity resolution comprising: Acquire a data set to be resolved, and divide entity tuples in the data set to be resolved into at least one disjoint partition, wherein the data set to be resolved includes at least two entity tuples; For any partition, all entity tuples in the partition are divided into N intervals, M consecutive intervals form a window, and all windows are processed serially in sequence, where N is greater than M, and both N and M are positive integers greater than zero; For any window, a thread block is allocated to each interval in the window, wherein the thread block is used to match the target tuple in the corresponding interval with the entity tuples in the partition other than the target tuple to obtain a matching result, and the thread blocks in the window are processed in parallel; If the thread block corresponding to any interval in the window is executed, the executed thread block is used to process the interval in the next window of the window, until all intervals are processed by the corresponding thread blocks to obtain all matching results, and all the matching results are used to guide the data set to be resolved to deduplicate the entity tuples therein.

[0007] In a second aspect, an embodiment of the present application provides an entity resolution parallel processing device, the entity resolution parallel processing device comprising: An entity partitioning module, used to obtain a data set to be resolved, and divide entity tuples in the data set to be resolved into at least one disjoint partition, wherein the data set to be resolved includes at least two entity tuples; An interval window module is used to divide all entity tuples in any partition into N intervals, form M consecutive intervals into a window, and process all windows serially in sequence, wherein N is greater than M, and both N and M are positive integers greater than zero; A thread allocation module, for allocating a thread block to each interval in any window, wherein the thread block is used to match the target tuple in the corresponding interval with the entity tuple in the partition other than the target tuple to obtain a matching result, and the thread blocks in the window are processed in parallel; The thread loop execution module is used to use the executed thread block to process the interval in the next window of the window if the thread block corresponding to any interval in the window is executed, until all intervals are processed by the corresponding thread blocks to obtain all matching results, and all the matching results are used to guide the data set to be resolved to deduplicate the entity tuples therein.

[0008] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the entity resolution parallel processing method as described in the first aspect when executing the computer program.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the entity resolution parallel processing method as described in the first aspect is implemented.

[0010] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application divides the entity tuples in the data set to be resolved into at least one non-overlapping partition, and for any partition, divides all tuples in the partition into N intervals, forms a window with M consecutive intervals, and processes all windows serially in sequence. For any window, a thread block is assigned to each partition in the window. If the thread block corresponding to any interval in the window is executed, the executed thread block is used to process the interval in the next window of the window, until all intervals are processed by the corresponding thread blocks and all matching results are obtained.

[0011] Among them, by adopting an asynchronous pipeline architecture, the tuples in the data set to be digested are divided into at least one non-overlapping partition, and each partition is further subdivided into multiple intervals to form windowed processing. The intervals in each window are processed in parallel by independent thread blocks, which optimizes resource utilization, ensures the workload balance of each thread block, and improves parallel efficiency, thereby accelerating the execution of parallel entity digestion tasks. At the same time, serial processing between windows also ensures the order and consistency of data, allows continuous transmission of processing results, and ensures the accuracy of processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0013] Figure 1 This is a schematic diagram of an application environment of an entity resolution parallel processing method provided in Example 1 of the present application; Figure 2 It is a flowchart of a parallel processing method for entity resolution provided in Example 2 of the present application; Figure 3 This is a schematic diagram of the interval and window division of a partition provided in Example 2 of the present application; Figure 4 It is a flowchart of a parallel processing method for entity resolution provided in Example 3 of the present application; Figure 5 This is a schematic diagram of the execution time of a thread block provided in Embodiment 3 of the present application; Figure 6 This is a schematic diagram of the execution time of a thread block cross-interval task stealing provided in the third embodiment of the present application; Figure 7 It is a flowchart of a parallel processing method for entity resolution provided in Example 4 of the present application; Figure 8 This is a schematic diagram of execution time of task stealing within a thread block interval provided by the fourth embodiment of the present application; Fig. 9 It is a flowchart of a parallel processing method for entity resolution provided in Example 5 of the present application; Fig.10 It is a flowchart of a parallel processing method for entity resolution provided in Example 6 of the present application; Fig.11 It is a structural diagram of a parallel processing device for entity resolution provided in Embodiment 7 of the present application; Fig.12 It is a structural diagram of a computer device provided in Example 8 of the present application. DETAILED DESCRIPTION

[0014] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0015] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0016] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0017] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0018] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0019] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0020] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0021] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0022] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0023] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0024] The first embodiment of the present application provides a parallel processing method for entity resolution, which can be applied in the following cases: Figure 1 In the application environment, the server communicates with the client, the server provides entity resolution parallel processing services, and the client triggers entity resolution parallel processing tasks to the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs), and other devices. The computer device corresponding to the server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0025] See also Figure 2 , is a flow chart of a parallel processing method for entity resolution provided in the second embodiment of the present application. The parallel processing method for entity resolution is applied to Figure 1 The server in the example connects to the client to obtain the dataset to be digested sent by the client. Figure 2 As shown, the entity resolution parallel processing method may include the following steps: Step S201: obtain a data set to be resolved, and divide entity tuples in the data set to be resolved into at least one disjoint partition.

[0026] In this embodiment, the data set to be decomposed may refer to the data set on which entity decomposition is to be performed, and the data set to be executed includes at least two tuples of record data information, each tuple consists of a single or multiple attributes and corresponding attribute values, the attribute may refer to a variable or field that describes the characteristics of the tuple, the attribute value may refer to the specific numerical value of the corresponding attribute, and the partition may refer to the non-overlapping data partitions obtained by dividing the entity tuples in the data set to be decomposed.

[0027] For example, given a pattern ,in, For an attribute, is the entity identifier (ID), each A tuple of represents an entity. The dataset to be digested is a set of A tuple of patterns.

[0028] Specifically, for any tuple in the data set to be digested, a hash key can be calculated according to the attributes of the tuple through a hash function, and tuples with the same hash key can be divided into the same partition.

[0029] Step S202: for any partition, all entity tuples in the partition are divided into N intervals, M consecutive intervals form a window, and all windows are processed serially in sequence.

[0030] Step S203: for any window, a thread block is allocated to each interval in the window.

[0031] Step S204, if the thread block corresponding to any interval in the window is executed, the executed thread block is used to process the interval in the next window of the window, until all intervals are processed by the corresponding thread blocks and all matching results are obtained.

[0032] In this embodiment, the interval may refer to a data interval obtained by dividing a partition, and the window may refer to a parallel sliding window (PSW), wherein N is greater than M, and both N and M are positive integers greater than zero. The thread block is used to match the target tuple in the corresponding interval with the entity tuples in the partition except the target tuple to obtain a matching result. The target tuple may refer to any tuple in the corresponding interval, and the matching result may refer to the result of entity decomposition of the target tuple with the entity tuples in the partition except the target tuple, that is, if the target tuple matches the tuple in the partition except the target tuple, it is determined that the two tuples represent the same real entity; if they do not match, it is determined that the two tuples represent different real entities. All matching results are used to guide the data set to be decomposed to deduplicate the entity tuples therein, that is, to determine the tuple pairs representing the same real entity in the data set to be decomposed, and the tuple pairs representing different real entities in the data set to be decomposed.

[0033] Specifically, for any grouping, all entity tuples in the partition are evenly divided into N intervals, and a window of preset length is used to slide on the N intervals with a step size of preset length. The interval included when each sliding stops is determined to be the interval of the corresponding window, and the window is used to slide and process the N intervals in sequence. For any window, a thread block is assigned to each interval in the window, and the thread blocks in the window are processed in parallel; when all threads in any thread block in the window are executed, the thread block writes the processing results back to the host memory, and then continues to process the interval in the next window until the window slides to the end of the partition, and all intervals are processed by the corresponding thread blocks to obtain all matching results.

[0034] For example, Figure 3 As shown, it is a schematic diagram of the interval and window division of a partition provided in the second embodiment of the present application.

[0035] All entity tuples in partition P are evenly divided into 9 intervals, namely interval 1, interval 2, interval 3, interval 4, interval 5, interval 6, interval 7, interval 8 and interval 9. The window size is 3, that is, 3 consecutive intervals form a window, and the 9 intervals are divided into 3 windows, namely sliding window 1, sliding window 2 and sliding window 3. Sliding window 1 includes interval 1, interval 2 and interval 3, sliding window 2 includes interval 4, interval 5 and interval 6, and sliding window 3 includes interval 7, interval 8 and interval 9. Slidingwindow 1, sliding window 2 and sliding window 3 are processed serially in sequence. The intervals in each window are assigned to three thread blocks (TBs), namely TB1, TB2 and TB3. For example, in Slidingwindow 1, interval 1 is assigned to TB1, and each thread in TB1 is responsible for matching the tuples in interval 1 with all other tuples in partition P to obtain matching results. Interval 2 is assigned to TB2, and each thread in TB2 is responsible for matching the tuples in interval 2 with all other tuples in partition P to obtain matching results. Interval 3 is assigned to TB3, and each thread in TB3 is responsible for matching the tuples in interval 3 with all other tuples in partition P to obtain matching results. TB1, TB2 and TB3 process interval 1, interval 2 and interval 3 in parallel. If all threads in TB1 within Sliding window 1 are executed, TB1 writes the processing results back to the host memory, and then TB1 continues to process interval 4 in Sliding window 2. After all threads in TB1 within Sliding window 2 are executed, TB1 writes the processing results back to the host memory, and then TB1 continues to process interval 7 in Sliding window 3. Correspondingly, TB2 processes interval 2, interval 5 and interval 8 in sequence, and TB3 processes interval 3, interval 6 and interval 9 in sequence, until all intervals are processed by the corresponding thread blocks and all matching results are obtained. The embodiment of the present application adopts an asynchronous pipeline architecture to divide the tuples in the data set to be digested into at least one non-overlapping partition, and each partition is further subdivided into multiple intervals to form windowed processing. The intervals within each window are processed in parallel by independent thread blocks, which optimizes resource utilization, ensures the workload balance of each thread block, and improves parallel efficiency, thereby accelerating the execution of parallel entity digestion tasks. At the same time, serial processing between windows also ensures the order and consistency of data, allows continuous transmission of processing results, and ensures the accuracy of processing.

[0036] See also Figure 4 , is a flow chart of a parallel processing method for entity resolution provided in Example 3 of the present application. Figure 4 As shown, after forming a window from M consecutive intervals in the above step S202, the following steps may be further included: Step S401, constructing a bitmap.

[0037] Step S402 : for any thread block, if the thread block is in an idle state, a section whose processing state is unprocessed is selected from the bitmap.

[0038] Step S403: Use the thread block to process the unprocessed interval to obtain the corresponding matching result.

[0039] In this embodiment, a bitmap is maintained in the global memory. Each bit in the bitmap represents the processing status of an interval being processed by the corresponding thread block. The thread block in the idle state processes the intervals allocated to all windows and completes the processing. For example, Figure 3 As shown, if TB1 processes interval 1, interval 4, and interval 7 assigned to all windows and completes the processing, TB1 is an idle thread block.

[0040] In the above second embodiment, for any window, a thread block is allocated to each interval in the window. For a partition, the number of intervals processed by each thread block is roughly equal. For example, Figure 3 As shown in the partition, three thread blocks are assigned three partitions, namely, TB1 processes interval 1, interval 4 and interval 7, TB2 processes interval 2, interval 5 and interval 8, and TB3 processes interval 3, interval 6 and interval 9. However, since the GPU adopts the single instruction multiple thread execution mode, when some threads execute for a long time, other threads may be idle, for example, Figure 5 , which is a schematic diagram of the execution time of a thread block provided in the third embodiment of the present application. Figure 3 For the partition shown in the figure, it takes 10 time units for TB1 to process interval 1, interval 4, and interval 7, 4 time units for TB2 to process interval 2, interval 5, and interval 8, and 3 time units for TB3 to process interval 3, interval 6, and interval 9. The total time consumption of TB1, TB2, and TB3 is inconsistent, resulting in 6 time units of idleness for TB2 and 7 time units of idleness for TB3.

[0041] In order to further balance the workload, a bitmap can be maintained in the global memory. For any thread block, when the thread block is executed, it first processes the intervals allocated to it by all windows. After processing and completing the intervals allocated to all windows, if the thread block is in an idle state, it can check the bitmap, steal an unprocessed interval from other thread blocks, and update the bitmap to avoid repeated processing. The stolen unprocessed interval is processed to obtain the corresponding matching result. This process can be called a cross-interval task stealing mechanism.

[0042] For example, Figure 6 , which is a schematic diagram of the execution time of a thread block cross-interval task stealing provided by the third embodiment of the present application. Figure 3 In the partition shown in the figure, after TB3 completes all window-assigned intervals after 3 time units, it steals interval 4 from TB1 by checking the bitmap. After TB2 completes all window-assigned intervals after 4 time units, it steals interval 7 from TB1 for processing by checking the bitmap. Through cross-interval task stealing, the total execution time is reduced from Figure 5 The 10 time units shown are reduced to 7 time units.

[0043] In the embodiment of the present application, after roughly equal tasks are assigned to threads through windows, during the execution process, a cross-interval task stealing mechanism is used to adaptively implement dynamic adjustment of tasks between different threads, thereby avoiding the situation where some thread blocks are idle after processing their own tasks while other thread blocks are still busy, further implementing load balancing, improving the overall utilization of hardware resources, and by reducing the waiting time and idle time of thread blocks, the total execution time is also reduced as a whole, thereby improving the efficiency of parallel processing.

[0044] See also Figure 7 , is a flow chart of a parallel processing method for entity resolution provided in Example 4 of the present application. Figure 7 As shown, before selecting an interval with a processing status of unprocessed from the bitmap in the above step S402, the following steps may also be included: Step S701, detecting whether there is an interval in the bitmap whose processing status is unprocessed.

[0045] Step S702: if it is detected that there is an unprocessed interval in the bitmap, then a process of selecting an unprocessed interval from the bitmap is executed.

[0046] Specifically, for any thread block, if the thread block is in an idle state, it is detected whether there is an interval with an unprocessed processing status in the bitmap. If it is detected that there is an interval with an unprocessed processing status in the bitmap, an interval with an unprocessed processing status is selected from the bitmap, and the thread block is used to process the unprocessed interval to obtain a corresponding matching result.

[0047] Step S703: If it is detected that there is no interval in the bitmap whose processing status is unprocessed, it is determined that the processing status is an interval in execution.

[0048] Step S704 , notifying the thread block corresponding to the executing interval to divide the remaining execution tasks corresponding to the executing interval into a preset number of subtasks.

[0049] Step S705 , leaving one of the subtasks in the thread block corresponding to the interval being executed for processing, and obtaining a corresponding matching result.

[0050] Step S706: Distribute each remaining subtask to an idle thread block for processing to obtain a corresponding matching result.

[0051] If it is detected that there is no interval in the bitmap whose processing status is unprocessed, the processing status is determined to be the executing interval, the idle number of the thread block in the idle state is obtained, and the idle number is sent as a notification to the thread block corresponding to the executing interval, the idle number is increased by 1 to obtain a preset number, and the remaining execution tasks corresponding to the executing interval are divided into a preset number of subtasks, one of the subtasks is left in the thread block corresponding to the executing interval for processing, and the corresponding matching result is obtained, and each of the remaining subtasks is distributed to an idle thread block for processing to obtain the corresponding matching result. This process can be called an intra-interval task stealing mechanism.

[0052] For example, Figure 8 , which is a schematic diagram of the execution time of task stealing within a thread block interval provided by the fourth embodiment of the present application. Figure 3 As shown in the partition, after TB3 completes the intervals assigned to all windows after 3 time units, it steals the task across intervals and steals interval 4 from TB1 for processing. After TB2 completes the intervals assigned to all windows after 4 time units, it steals the task across intervals and steals interval 7 from TB1 for processing. After TB3 completes the processing of the stolen interval 4, it checks that there is no interval in the bitmap with the processing status of unprocessed, and then determines that the processing status is the interval interval 7 in execution. At this time, the number of idle thread blocks in the idle state is 1, that is, TB3, and the preset number of copies is 2. The remaining tasks corresponding to the executing interval interval 7 are divided into 2 subtasks, one subtask is continued to be processed by TB2, and the other subtask is processed by TB3. Through cross-interval task stealing and intra-interval task stealing, the total execution time is reduced from Figure 5 The 10 time units shown are reduced to 6 time units.

[0053] In the embodiment of the present application, after roughly equal tasks are assigned to threads through windows, during the execution process, dynamic adjustment of tasks between different threads is adaptively achieved through a cross-interval task stealing mechanism and an intra-interval task stealing mechanism, thereby avoiding the situation where some thread blocks are idle after processing their own tasks while other thread blocks are still busy, further achieving load balancing, improving the overall utilization of hardware resources, and by reducing the waiting time and idle time of thread blocks, the total execution time is also reduced as a whole, thereby improving the efficiency of parallel processing.

[0054] See also Fig. 9 , is a flow chart of a parallel processing method for entity resolution provided in Example 5 of the present application, such as Fig. 9As shown, in the above step S201, the entity tuples in the to-be-resolved data set are divided into at least one disjoint partition, which may include the following steps: Step S901, obtaining an execution plan of the data set to be resolved, determining S equal sign predicates in the execution plan that are not reused by at least two rules, and obtaining S hash functions according to the S equal sign predicates.

[0055] Step S902: for any entity tuple in the data set to be resolved, call any hash function from the S hash functions to perform calculation to obtain the corresponding partition key.

[0056] Step S903, traverse all entity tuples to obtain partition keys of all entity tuples, and divide the entity tuples in the data set to be resolved according to the partition keys to obtain at least one disjoint partition.

[0057] In this embodiment, the dataset to be resolved can be defined as a set of tuples with R pattern (refer to the content in step S201). Then, when entity resolution is performed on the dataset to be resolved based on the rule, the rule can be defined on the pattern R as: ,in, It is about two tuples and The set of predicates, express , For rules The prerequisite is is the result, when the tuple and tuples Satisfy the rules The prerequisite is that the tuple and tuples Represents the same entity. The predicate in can be defined as ,in, and is a property, c is a constant, and Used to compare whether the A attribute value of tuple t is equal to the B attribute value of tuple s. It is used to compare the similarity between the attribute value of A of tuple t and the attribute value of B of tuple s. The similarity can be measured using various similarity metrics, such as edit distance or Jaccard similarity.

[0058] The execution plan specifies the order of evaluation of rules and predicates in the rule set for entity resolution of the dataset to be resolved. The equal sign predicate can refer to the predicate for equality comparison. For example, and ,The partition key may refer to a hash value calculated by a hash function, which is used to determine to which partition a tuple should be assigned.

[0059] Specifically, an execution plan can be generated by a database query optimizer based on the rule set discovered offline, and the evaluation order of the rules and predicates in the execution rule set can be executed. After obtaining the execution plan, S equal sign predicates that are not reused in the execution plan are determined, and S hash functions are obtained based on the S equal sign predicates. For any tuple in the data set to be digested, an identifier (Identification, ID) can be assigned to the tuple to indicate the hash function used. According to the ID of the tuple, the corresponding hash function is called from the S hash functions for calculation to obtain the corresponding partition key, and all tuples in the data set to be digested are traversed to obtain the partition keys of all tuples. According to the partition key, the entity tuples in the data set to be digested are divided to obtain at least one non-intersecting partition, and only tuples sharing the same hash function will be divided into the same partition.

[0060] Optionally, if data partitioning is done by similarity predicates in the execution plan (e.g. ) is completed by the hash function, then tuples may appear and tuples represents the same real entity, but and Located in different partitions, in this case, the partition with local data The kernel can choose to pull partitions from another kernel ,like and If they are on the same GPU, the pull operation is performed directly. and Pull to the same partition, no data transmission is involved. and If they are located on different GPUs, you can call the corresponding function to pull the corresponding tuples from the corresponding GPU to the same partition.

[0061] In an embodiment of the present application, according to S equal sign predicates that are not reused by at least two rules in the execution plan, S hash functions are used to divide the tuples in the data set to be resolved, and at least one non-intersecting partition is obtained. The partitioning strategy provided can make the S hash functions selective according to the execution plan generated by the database optimizer, and may be shared by multiple rules, so that efficient partitioning can be achieved with a small number of hash functions, and an ID is assigned to each tuple to indicate the hash function used. Only tuples sharing the same hash function will be compared, thereby reducing redundant calculations caused by multiple hash functions.

[0062] See also Fig.10 , is a flow chart of a parallel processing method for entity resolution provided in Example 6 of the present application, such as Fig.10 As shown, after the entity tuples in the to-be-resolved data set are divided into at least one disjoint partition in the above step S201, the following steps may also be included: Step S1001, randomly hashing and assigning a GPU to each partition, for any partition, if the GPU corresponding to the partition is available, all entity tuples in the partition are divided into N intervals.

[0063] Step S1002: If the GPU corresponding to the partition is unavailable, the partition is routed to the GPU corresponding to other partitions through a preset direction, and all entity tuples in the partition are divided into N intervals.

[0064] Specifically, when the entity resolution task is extended to multiple GPUs for processing, the partitions obtained by dividing the data set to be resolved and the multiple GPUs can be randomly hashed to positions on the unit circle. For any partition, if the GPU corresponding to the partition is available, all entity tuples in the partition are divided into N intervals (for details, please refer to the contents in steps S202 to 204). If the partition is assigned to an unavailable GPU (the GPU has no idle cores or no available PCIe (PCI Express) bandwidth), it can be rerouted in a clockwise direction to the nearest available GPU, and all entity tuples in the partition are divided into N intervals (for details, please refer to the contents in steps S202 to 204).

[0065] In the embodiment of the present application, when the entity resolution task is extended to multiple GPUs for processing, a GPU is randomly hashed to each partition. For any partition, if the GPU corresponding to the partition is available, all entity tuples in the partition are divided into N intervals. If the GPU corresponding to the partition is unavailable, it is routed to the GPU corresponding to other partitions through a preset direction, and all entity tuples in the partition are divided into N intervals. Load balancing of multiple GPUs is achieved, and resource contention is reduced.

[0066] Corresponding to the entity resolution parallel processing method of the above embodiment, Fig.11 The structure block diagram of a parallel processing device for entity decomposition provided in the seventh embodiment of the present application is shown. The parallel processing device for entity decomposition is applied to Figure 1 For ease of description, only the parts related to the embodiment of the present application are shown.

[0067] See also Fig.11 , the entity resolution parallel processing device comprises: The entity partitioning module 1101 is used to obtain a data set to be resolved, and divide the entity tuples in the data set to be resolved into at least one disjoint partition, wherein the data set to be resolved includes at least two entity tuples; The interval window module 1102 is used for dividing all entity tuples in any partition into N intervals, forming a window with M consecutive intervals, and processing all windows serially in sequence, wherein N is greater than M, and both N and M are positive integers greater than zero; The thread allocation module 1103 is used for allocating a thread block to each interval in any window, wherein the thread block is used to match the target tuple in the corresponding interval with the entity tuple in the partition other than the target tuple to obtain a matching result, and the thread blocks in the window are processed in parallel; The thread loop execution module 1104 is used to use the executed thread block to process the interval in the next window of the window if the thread block corresponding to any interval in the window is executed, until all intervals are processed by the corresponding thread blocks to obtain all matching results, and all the matching results are used to guide the data set to be resolved to deduplicate the entity tuples therein.

[0068] Optionally, the interval window module 1102 includes: A sliding window unit, used to slide a window of a preset length on the N intervals with a step length of the preset length, wherein the preset length is the length of M consecutive intervals; The interval determining unit is used to determine that the interval included in each sliding stop is the interval of the corresponding window.

[0069] Optionally, the entity resolution parallel processing device further includes: A bitmap construction module, used for constructing a bitmap, each bit in the bitmap represents a processing state of an interval being processed by a corresponding thread block; A cross-interval task stealing module is used for selecting an interval with an unprocessed processing status from the bitmap for any thread block if the thread block is in an idle state, wherein the thread block in the idle state processes the intervals allocated to all windows and completes the processing; The cross-interval task processing module is used to use the thread block to process the unprocessed interval to obtain a corresponding matching result.

[0070] Optionally, the entity resolution parallel processing device further includes: A detection module, used for detecting whether there is an interval in the bitmap whose processing status is unprocessed; A first detection and determination module, configured to, if it is detected that there is an interval in the bitmap with a processing status of unprocessed, execute the step of selecting an interval in the bitmap with a processing status of unprocessed; A second detection and determination module is used to determine that the processing status is an executing interval if it is detected that there is no interval in the bitmap whose processing status is unprocessed; A first task division module, used for notifying the thread block corresponding to the executing interval, and dividing the remaining execution task corresponding to the executing interval into a preset number of subtasks; A task processing module, used for leaving one of the subtasks in the thread block corresponding to the interval being executed for processing, and obtaining a corresponding matching result; The intra-interval task stealing module is used to distribute each remaining subtask to an idle thread block for processing to obtain the corresponding matching result.

[0071] Optionally, the entity resolution parallel processing device further includes: The idle acquisition module is used to obtain the idle number of all thread blocks in the idle state; The notifying the thread block corresponding to the executing interval and dividing the remaining execution task corresponding to the executing interval into a preset number of subtasks includes: An idle sending unit, configured to send the idle number as a notification to a thread block corresponding to the executing interval; The second task division unit is used to add 1 to the idle number to obtain a preset number of copies, and divide the remaining execution tasks corresponding to the execution interval into subtasks of the preset number of copies.

[0072] Optionally, the physical partitioning module 1101 includes: A predicate determination unit, configured to obtain an execution plan of the data set to be resolved, determine S equal sign predicates in the execution plan that are not reused by at least two rules, and obtain S hash functions according to the S equal sign predicates; A hash calculation unit, used for calling any hash function from the S hash functions to calculate any entity tuple in the data set to be resolved, to obtain a corresponding partition key; The hash partitioning unit is used to traverse all entity tuples to obtain partition keys of all entity tuples, and to partition the entity tuples in the to-be-resolved data set according to the partition keys to obtain at least one disjoint partition.

[0073] Optionally, the entity resolution parallel processing device further includes: A first GPU scheduling module is used to randomly hash and assign a GPU to each partition, and for any partition, if the GPU corresponding to the partition is available, divide all entity tuples in the partition into N intervals; The second GPU scheduling module is used for routing to GPUs corresponding to other partitions through a preset direction if the GPU corresponding to the partition is unavailable, and executing the step of dividing all entity tuples in the partition into N intervals.

[0074] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0075] Fig.12 This is a schematic diagram of the structure of a computer device provided in Example 8 of the present application. Fig.12 As shown, the computer device of this embodiment includes: at least one processor ( Fig.12 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned entity resolution parallel processing method embodiments are implemented.

[0076] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Fig.12 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0077] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0078] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0079] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0080] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing the computer program product.

[0081] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0083] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A parallel processing method for entity resolution, characterized in that: The entity resolution parallel processing method comprises: Acquire a data set to be resolved, and divide entity tuples in the data set to be resolved into at least one disjoint partition, wherein the data set to be resolved includes at least two entity tuples; For any partition, all entity tuples in the partition are divided into N intervals, M consecutive intervals form a window, and all windows are processed serially in sequence, where N is greater than M, and both N and M are positive integers greater than zero; For any window, a thread block is allocated to each interval in the window, wherein the thread block is used to match the target tuple in the corresponding interval with the entity tuples in the partition other than the target tuple to obtain a matching result, and the thread blocks in the window are processed in parallel; If the thread block corresponding to any interval in the window is executed, the executed thread block is used to process the interval in the next window of the window, until all intervals are processed by the corresponding thread blocks to obtain all matching results, and all the matching results are used to guide the data set to be resolved to deduplicate the entity tuples therein.

2. The entity resolution parallel processing method according to claim 1, characterized in that: The forming of M consecutive intervals into a window includes: Using a window of a preset length, sliding on the N intervals with a step length of the preset length, the preset length being the length of M consecutive intervals; It is determined that the interval included in each sliding stop is the interval of the corresponding window.

3. The entity resolution parallel processing method according to claim 1, characterized in that: After forming the M consecutive intervals into a window, the method further includes: Construct a bitmap, each bit in the bitmap represents a processing state of an interval being processed by a corresponding thread block; For any thread block, if the thread block is in an idle state, an interval whose processing state is unprocessed is selected from the bitmap, wherein the thread block in the idle state processes the intervals allocated to all windows and the processing is completed; The unprocessed interval is processed using the thread block to obtain a corresponding matching result.

4. The entity resolution parallel processing method according to claim 3, characterized in that: Before selecting an interval whose processing status is unprocessed from the bitmap, the method further includes: Detect whether there is an interval in the bitmap whose processing status is unprocessed; If it is detected that there is an interval in the bitmap whose processing status is unprocessed, selecting an interval in the bitmap whose processing status is unprocessed is executed; If it is detected that there is no interval in the bitmap whose processing status is unprocessed, determining that the processing status is an interval in execution; Notify the thread block corresponding to the executing interval, and divide the remaining execution task corresponding to the executing interval into a preset number of subtasks; leaving one of the subtasks in the thread block corresponding to the interval being executed for processing, and obtaining a corresponding matching result; Each of the remaining subtasks is distributed to an idle thread block for processing to obtain the corresponding matching result.

5. The entity resolution parallel processing method according to claim 4, characterized in that: Before notifying the thread block corresponding to the executing interval, the method further includes: Get the number of idle thread blocks in all idle states; The notifying the thread block corresponding to the executing interval and dividing the remaining execution task corresponding to the executing interval into a preset number of subtasks includes: Sending the idle number as a notification to the thread block corresponding to the executing interval; The idle number is increased by 1 to obtain a preset number of copies, and the remaining execution tasks corresponding to the execution interval are divided into subtasks of the preset number of copies.

6. The entity resolution parallel processing method according to claim 1, characterized in that: The step of dividing the entity tuples in the to-be-resolved data set into at least one disjoint partition comprises: Obtain an execution plan for the data set to be resolved, determine S equal sign predicates in the execution plan that are not reused by at least two rules, and obtain S hash functions according to the S equal sign predicates; For any entity tuple in the data set to be resolved, call any hash function from the S hash functions to perform calculation to obtain a corresponding partition key; All entity tuples are traversed to obtain partition keys of all entity tuples, and the entity tuples in the to-be-resolved data set are divided according to the partition keys to obtain at least one disjoint partition.

7. The entity resolution parallel processing method according to any one of claims 1 to 6, characterized in that: After dividing the entity tuples in the to-be-resolved data set into at least one disjoint partition, the method further includes: A GPU is randomly assigned to each partition by hashing. For any partition, if the GPU corresponding to the partition is available, all entity tuples in the partition are divided into N intervals; If the GPU corresponding to the partition is unavailable, it is routed to the GPU corresponding to other partitions through a preset direction, and all entity tuples in the partition are divided into N intervals.

8. An entity resolution parallel processing device, characterized in that: The entity resolution parallel processing device comprises: An entity partitioning module, used to obtain a data set to be resolved, and divide entity tuples in the data set to be resolved into at least one disjoint partition, wherein the data set to be resolved includes at least two entity tuples; An interval window module is used to divide all entity tuples in any partition into N intervals, form M consecutive intervals into a window, and process all windows serially in sequence, wherein N is greater than M, and both N and M are positive integers greater than zero; A thread allocation module, for allocating a thread block to each interval in any window, wherein the thread block is used to match the target tuple in the corresponding interval with the entity tuple in the partition other than the target tuple to obtain a matching result, and the thread blocks in the window are processed in parallel; The thread loop execution module is used to use the executed thread block to process the interval in the next window of the window if the thread block corresponding to any interval in the window is executed, until all intervals are processed by the corresponding thread blocks to obtain all matching results, and all the matching results are used to guide the data set to be resolved to deduplicate the entity tuples therein.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the entity resolution parallel processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the entity resolution parallel processing method according to any one of claims 1 to 7 is implemented.

Images