Optimization control method, device and equipment for single-machine diagram data cleaning and medium

Abstract

The invention relates to an optimization control method, device and equipment for single machine diagram data cleaning and a medium. The method comprises the following steps: acquiring target graph data and a rule set, disassembling all double-star mode rules in the rule set into single-star components, determining a component structure of each single-star component, carrying out intersection and union operation on the single-star components of which the component structures comprise the same nodes and edges to obtain a single-star beam comprising a common substructure and a complete structure, and on the basis of each single star beam, matching a corresponding matching entity from the target map data, performing enumeration verification on all the matching entities by using the rule set to obtain a conflicting matching entity pair, and performing conflict repair on the conflicting matching entity pair to obtain a map data cleaning result. According to the method, the double-star mode rule is disassembled, the single-star component obtained through disassembly is subjected to single-star beam construction, entity matching is executed through the single-star beam, a shared structure does not need to be calculated repeatedly, redundant calculation in the mode matching stage is effectively avoided, and the execution efficiency of the system is improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and particularly to an optimized control method, device, equipment, and medium for single-machine graph data cleaning. Background Art

[0002] In recent years, with the advancement of the social informatization process, the scale of graph data generated or collected by computer systems has been rapidly expanding. For example, Facebook's social network has nearly 2 billion users, and when modeled as a graph, it will contain nearly 2 billion nodes and an even higher number of edges, which poses a severe challenge to the performance of graph data cleaning systems. However, in some specific scenarios, such as industrial Internet of Things, computing devices are usually deployed at the edge, the deployment of distributed clusters is difficult to achieve, and the cloud-based computing model faces problems such as cost and privacy, resulting in an increasing demand for single-machine systems.

[0003] Considering the actual requirements for large-scale graph data cleaning and resource-constrained usage scenarios, various related solutions have been proposed in recent years, which can be roughly classified into single-machine general graph computing systems, machine learning-based graph cleaning systems, and graph data cleaning systems based on graph cleaning rules. Among them, single-machine general graph computing systems usually rely on a single parallel model to optimize the running efficiency of specific types of graph computing tasks (e.g., connected component calculation, shortest path query). However, for more complex graph cleaning tasks, such systems cannot fully utilize heterogeneous computing resources and often suffer from problems such as resource idleness and GPU warp divergence, resulting in impaired overall performance; machine learning-based graph data cleaning systems rely on a large amount of labeled data and model hyperparameter tuning, and their prediction results do not guarantee correctness and interpretability, so they are not suitable for scenarios with insufficient data annotation or high requirements for result reliability; graph cleaning systems based on graph cleaning rules need to apply all the rules in the graph cleaning rule set to the data graph, which involves frequent graph pattern matching and pairwise verification of graph pattern entities, with a large computational overhead. In addition, due to the overlap of patterns and attributes between the input rules, sequential application of rules will lead to a large amount of redundant calculations.

[0004] Single-machine graph data cleaning systems are rule-based graph data cleaning that involves a large number of computationally intensive tasks such as graph pattern matching and entity pair enumeration. The limited computational resources of existing single-machine systems are difficult to support large-scale graph data cleaning tasks. Graph data cleaning requires the derivation of all the rules in the input graph cleaning rule set. This process involves a large amount of repeated calculations, further exacerbating the performance bottleneck. Due to the use of a single parallel model, such as a vertex / edge / subgraph-centered parallel model, these parallel models have problems such as resource idleness and unsaturated parallelism and cannot fully utilize computational resources.

[0005] Therefore, how to optimize the processing process of single-machine graph data cleaning to improve the computing efficiency has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, the embodiments of the present application provide an optimization control method, device, equipment and medium for single-machine graph data cleaning to solve the problem of how to optimize the processing process of single-machine graph data cleaning to improve the computing efficiency.

[0007] In a first aspect, the embodiments of the present application provide an optimization control method for single-machine graph data cleaning, including: Obtaining target graph data and a rule set composed of double-star pattern rules obtained by rule discovery based on the target graph data, where the double-star pattern rules represent the association and dependence relationships between any two different entities; Decomposing all double-star pattern rules in the rule set into single-star components, and determining the component structure of each single-star component. The single-star component includes a predicate condition and a materialized property that an entity needs to satisfy, and the component structure includes at least one node and an edge connecting two nodes; Performing intersection operation and union operation on single-star components with the same node and edge in the component structure to obtain single-star bundles including common sub-structures and complete structures. The common sub-structure includes shared node connection relationships and shared predicate conditions, and the complete structure includes all node connection relationships and all materialized properties of all merged single-star components; Based on each single-star bundle, matching corresponding matching entities from the target graph data, using the rule set to enumerate and verify all matching entities, obtaining conflicting matching entity pairs, and repairing the conflicting matching entity pairs to obtain the graph data cleaning result.

[0008] In a second aspect, an optimization control device for single-machine graph data cleaning according to the embodiments of the present application includes: A data acquisition module, configured to acquire target graph data and a rule set composed of double-star pattern rules obtained by rule discovery based on the target graph data, where the double-star pattern rules represent the association and dependence relationships between any two different entities; A pattern decomposition module, configured to decompose all double-star pattern rules in the rule set into single-star components, and determine the component structure of each single-star component. The single-star component includes a predicate condition and a materialized property that an entity needs to satisfy, and the component structure includes at least one node and an edge connecting two nodes; A pattern combination module is used to perform intersection and union operations on single-star components with component structures including the same nodes and edges, so as to obtain a single-star bundle including a common sub-structure and a complete structure. The common sub-structure includes shared node connection relationships and shared predicate conditions, and the complete structure includes all node connection relationships of all merged single-star components and all to-be-materialized attributes; A graph data cleaning module is used to, based on each single-star bundle, match corresponding matching entities from the target graph data, use the rule set to enumerate and verify all the matching entities, obtain conflicting matching entity pairs, and perform conflict repair on the conflicting matching entity pairs to obtain a graph data cleaning result.

[0009] In a third aspect, an embodiment of the present application provides a computer device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the optimized control method for single-machine graph data cleaning as described in the first aspect.

[0010] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the optimized control method for single-machine graph data cleaning as described in the first aspect.

[0011] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: The present application obtains the target graph data and a rule set composed of double-star pattern rules obtained by rule discovery based on the target graph data, disassembles all the double-star pattern rules in the rule set into single-star components, determines the component structure of each single-star component, performs intersection and union operations on single-star components with component structures including the same nodes and edges to obtain a single-star bundle including a common sub-structure and a complete structure, based on each single-star bundle, matches corresponding matching entities from the target graph data, uses the rule set to enumerate and verify all the matching entities to obtain conflicting matching entity pairs, and performs conflict repair on the conflicting matching entity pairs to obtain a graph data cleaning result. By disassembling the double-star pattern rules and constructing single-star bundles for the disassembled single-star components, and using the single-star bundles to execute matching entities, there is no need to repeatedly calculate the shared structure, effectively avoiding redundant calculations in the pattern matching stage and improving the execution efficiency of the system. Brief Description of the Drawings

[0012] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for use in the embodiments or the description of the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0013] Figure 1 It is a schematic diagram of an application environment of an optimized control method for single-machine graph data cleaning provided in Embodiment 1 of the present application; Figure 2 It is a schematic flowchart of an optimized control method for single-machine graph data cleaning provided in Embodiment 2 of the present application; Figure 3 It is an example diagram of graph data and rules in graph data cleaning provided in Embodiment 2 of the present application; Figure 4 It is a schematic flowchart of a graph data cleaning workflow provided in Embodiment 2 of the present application; Figure 5 It is an example diagram of data of a single-star component combination provided in Embodiment 2 of the present application; Figure 6 It is a schematic diagram of the architecture of a single-machine graph data cleaning system provided in Embodiment 2 of the present application; Figure 7 It is a schematic flowchart of an optimized control method for single-machine graph data cleaning provided in Embodiment 3 of the present application; [[ID=!23]] Figure 8 It is a schematic diagram of the structure of an optimized control device for single-machine graph data cleaning provided in Embodiment 4 of the present application; Figure 9 It is a schematic diagram of the structure of a computer device provided in Embodiment 5 of the present application. Detailed Description of the Invention

[0014] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are proposed to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0015] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0016] It should also be understood that the term "and / or" as used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0017] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0018] In addition, in the description of the specification and appended claims of this application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0019] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0020] It should be understood that the magnitudes of the sequence numbers of the steps in the following embodiments do not mean the order of execution is prior or posterior, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this application.

[0021] In order to illustrate the technical solution of this application, the following specific embodiments are used for illustration.

[0022] An optimized control method for single-machine graph data cleaning provided in Embodiment 1 of this application can be applied in an application environment such as Figure 1 where the client communicates with the server, the optimized control method is executed on the server, and the client sends a corresponding start instruction to the server to start graph cleaning rule matching. Graph data can be stored in the server, and the graph data can be directly called when performing graph cleaning rule matching. The server is configured with a storage disk, and the disk is used to provide memory support for graph cleaning rule matching.

[0023] Among them, the client includes but is not limited to computer devices such as palm computers, desktop computers, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, personal digital assistants (PDAs), etc. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0024] See Figure 2 , which is a schematic flowchart of an optimized control method for single-machine graph data cleaning provided in the second embodiment of this application. The above-mentioned optimized control method for single-machine graph data cleaning can be applied to Figure 1 the server in, and the server performs optimized control during graph cleaning rule matching, specifically for the optimized control of data processing during the graph cleaning rule matching process.

[0025] For example Figure 2 As shown, the optimized control method for single-machine graph data cleaning may include the following steps: Step S201: Obtain the target graph data and a rule set composed of double-star pattern rules obtained by rule discovery based on the target graph data.

[0026] In this embodiment, the double-star pattern rule represents the association and dependency relationship between any two different entities. The target graph data can be stored on the disk for later invocation. Correspondingly, the rule set can also be stored on the disk. Of course, the rules in the rule set can be discovered from the target graph data.

[0027] The process of graph cleaning rule matching in this application is as follows: For example Figure 3 As shown, it is an example diagram of graph data and rules in graph data cleaning provided in the second embodiment of this application. The mathematical form of the graph cleaning rule can be expressed as: , where is a double-star graph pattern used to identify related entities in the graph, X is a conjunction of a set of predicates, and p0 is a predicate; can represent the association and dependency relationship between entities in the graph. Among them, the rule will first match the double-star graph pattern in the graph G, and then assert that for each entity in the double-star graph pattern, if the machine learning model M sIt is considered that if the conferences of two paper nodes are the same as the author's name and the category of one of the papers is CS, then the category of the other paper should also be CS. This mathematical form based on logical reasoning ensures the correctness and interpretability of the graph data cleaning results. In addition, the graph cleaning rules are also compatible with machine learning models in the form of predicates. Therefore, the data cleaning results also have strong generalization ability. Due to its strong theoretical guarantee, more and more graph data cleaning systems choose graph cleaning rules as the theoretical basis.

[0028] As Figure 4 shown, the graph data cleaning based on graph cleaning rules in the embodiments of the present application includes three stages: (1) In the rule discovery stage, graph cleaning rules are enumerated according to user configuration and the enumerated rules are applied to the data graph samples input by the user. If the support and confidence of the enumerated rules are both higher than the user-set threshold, they will be placed in the output set Σ. When all enumerated rules are applied, the rule set Σ will be output as the result. (2) The error detection stage accepts the graph G and the output Σ of the previous stage as inputs and applies these rules to the data graph: First, the double-star pattern defined by the graph cleaning rules is matched in the graph, and then the dependency conditions defined by the rules are verified on the matching entities. Entities that do not meet the dependencies will be output as conflicts. (3) The error repair stage accepts the graph G, the rule set Σ, and the conflict data of the previous stage as inputs, repairs the errors in the graph data, and returns to the second stage to perform error detection again. When there is no conflict data output in the second stage, the process terminates.

[0029] In the above workflow, the matching of graph cleaning rules runs through the graph data cleaning process. Therefore, the optimization of the matching of graph cleaning rules is crucial for the overall performance optimization of the graph data cleaning system.

[0030] Step S202: Decompose all double-star pattern rules in the rule set into single-star components and determine the component structure of each single-star component.

[0031] Among them, the double-star pattern rule is a rule for two entities, while the single-star component is a rule for one entity. The double-star pattern rule can be decomposed into two single-star components. There is a shared substructure in the single-star components. Therefore, this shared substructure can be processed once instead of multiple times.

[0032] Decompose the double-star pattern in the rule into several independent single-star components , where is a single-star graph pattern, X defines the predicate condition that this single-star component needs to meet, F QDefines the attributes that the single-star component needs to materialize. Single-star components with shared substructures can be combined into a single-star bundle and matched as a whole, thus avoiding duplicate matching of the shared substructures. Therefore, the single-star component includes a predicate condition that an entity needs to satisfy and the attributes to be materialized, and the component structure includes at least one node and an edge connecting two nodes.

[0033] In this embodiment, the matching of the graph cleaning rule includes two stages: (1) Single-star graph pattern matching. In the graph cleaning task, there are shared substructures among the rules in the rule set (for example: shared star-shaped subpatterns, shared predicates). In this stage, the double-star pattern of the graph cleaning rule will be disassembled into two single-star components for separate matching, which can avoid excessive intermediate results caused by directly materializing the double-star pattern. (2) Double-star matching enumeration. In this stage, pairs will be enumerated one by one and it will be checked whether the matching of the two single-star patterns corresponding to the graph cleaning rule satisfies the predicate dependencies defined by the rule, and entity pairs that do not satisfy the dependencies will be regarded as data conflicts.

[0034] Disassemble each graph cleaning rule into two single-star components, then combine the single-star components with shared substructures into a single-star bundle, and finally match with the single-star bundle as a whole. As Figure 5 shown, it is a data example diagram of the combination of single-star components provided by the second embodiment of the present application. In this example, the double-star pattern of the rule can be disassembled into and two single-star components, and these two single-star components share three nodes x0, x1, x2 and two edges x0→x1, x0→x2.

[0035] Step S203, perform intersection operation and union operation on single-star components whose component structures include the same nodes and edges to obtain a single-star bundle including a common substructure and a complete structure.

[0036] Among them, the common substructure includes the shared node connection relationship and the shared predicate condition, and the complete structure includes all the node connection relationships and all the attributes to be materialized of all the combined single-star components.

[0037] As Figure 5 shown, extract the common substructure through the intersection operator ⊗, including: the shared subpattern Q ( l*r ) and the shared predicate X l ∩ X r . This part of the common substructure can be regarded as a filter. Any single-star match that does not satisfy the shared subpattern or the shared predicate cannot be a match of any single-star component that makes up the single-star bundle, so it can be safely filtered.

[0038] For entities that meet the filtering conditions defined by the common substructure, the union operator ⊕ is used to extract the complete structure, including the complete graph pattern Q (l+r) and the complete set of predicates defined by F Q(l+r) and materialize the entity with this. This materialization result can be regarded as a superset of all single-star pattern entities that make up a single-star bundle, so it can be equivalently regarded as the matching of any single-star component that makes up the single-star bundle.

[0039] Step S204: Based on each single-star bundle, match the corresponding matching entities from the target graph data, use the rule set to enumerate and verify all the matching entities, obtain the conflicting matching entity pairs, and repair the conflicts of the conflicting matching entity pairs to obtain the graph data cleaning result.

[0040] Among them, for any single-star bundle, matching the corresponding matching entities from the target graph data, that is, the target data in the target graph data. For each matching entity, use the two-star pattern rules in the above rule set to enumerate and pair any two matching entities. If there is a conflict, it means that the matching entity pair of the quantity is a conflicting matching entity pair. Repairing the conflicts of the conflicting matching entity pairs can obtain the graph data cleaning result.

[0041] In the embodiments of the present application, CPU / GPU heterogeneous computing resources are used. In order to make full use of CPU / GPU heterogeneous computing resources, the present application proposes a hybrid parallel model for data cleaning tasks, including: pipeline parallelism, single instruction multiple data (i.e., SIMD) parallelism, and independent task parallelism.

[0042] Among them, the pipeline system architecture: Such as Figure 6 shown, is the architecture schematic diagram of a single-machine graph data cleaning system provided in the second embodiment of the present application. The present invention divides the graph data cleaning task into two stages as follows: 1. Single-star bundle matching phase: (a) The data loader loads the graph data from the SSD into the memory; (b) The query parser parses the query parameters input by the user. If the task type is rule discovery, the rule generator is used to enumerate and generate a rule set. Otherwise, the rule parser loads and parses the rule set; (c) Subsequently, the pattern combiner reorganizes the rules in the rule set into a single-star bundle; (d) And the pattern matcher matches the single-star bundle entities in the data graph; (e) To reduce the computational complexity of subsequent matching enumeration, the matched entities are bucketed by the entity bucketizer according to their materialized attributes. Since there are differences in attributes among cross-bucket entities and it is impossible to form a pair of double-star patterns, the subsequent double-star matching phase can only enumerate the entities located in the same bucket; (f) For large-scale data graphs, the matched entities may not be able to be fully placed in the memory. Therefore, the embodiment of the present application introduces a buffer manager to manage the matched single-star bundle entities, and the part exceeding the memory will be swapped into the swap area in the SSD; (g) Finally, the matched entities in each bucket are submitted to the GPU via the GPU proxy as the input for the next stage task.

[0043] 2. Double-star matching enumeration phase: (a) The GPU proxy first loads the single-star bundle entities matched in the previous stage into the GPU video memory; (b) Each graph cleaning rule in the rule set corresponds to an enumeration task. The double-star matching enumerator assigns thread blocks to each enumeration task, and then checks whether each pair of single-star bundle entities satisfies the dependencies specified by the graph cleaning rule in the form of single instruction multiple data. All entity pairs that violate the rule dependencies will be marked as conflict data; (c) When all single-star bundle entity pairs have completed enumeration, the conflict data is copied back to the memory via the GPU proxy; (d) Finally, the result aggregator repairs the data conflicts and finally saves the result to the SSD.

[0044] Pipeline parallelism: Considering that there are a large number of branch judgments in the pattern matching of the first stage, in the parallel mode of single instruction multiple data, it is easy to cause warp divergence. Therefore, this stage uses CPU multi-core parallel acceleration; while the computational logic of the second stage is relatively regular, but it involves enumeration with O(N^2) complexity and has a large computational overhead. Therefore, GPU acceleration is used. The present invention effectively improves the system performance by overlapping the CPU calculations, GPU calculations, and data transfer overheads between the two stages in a pipeline manner.

[0045] A hybrid parallel model is designed for the graph data cleaning task. By pipeline parallelism between the CPU and GPU, the calculations of each stage algorithm and the data transfer overheads between the CPU and GPU are effectively overlapped; by single instruction multiple data parallelism, the powerful parallel capabilities of the GPU are effectively utilized; and by dependency-free parallelism, the resource utilization rate of the GPU is further improved.

[0046] Optionally, after matching corresponding matching entities from the target graph data based on each single-star beam, the method further includes: Bucketing all the matching entities according to the materialization attributes to obtain N entity buckets, where the matching entities with the same materialization attributes are in the same entity bucket, and N is an integer greater than zero; Using the rule set to perform enumeration verification on all the matching entities to obtain conflicting matching entity pairs, including: For any entity bucket, using the rule set to perform enumeration verification on all the matching entities in the entity bucket to obtain conflicting matching entity pairs corresponding to the entity bucket; Traversing all the entity buckets to obtain all the conflicting matching entity pairs.

[0047] Among them, setting up entity buckets can group matching entities with the same materialization attributes in one bucket. For the materialized entities in an entity bucket, they can be paired by the double-star mode rules. However, cross-bucket entities have differences in attributes and cannot form a pair of double-star modes. Therefore, in the subsequent double-star matching stage, only the entities in the same bucket need to be enumerated.

[0048] Optionally, using a GPU to execute the enumeration task, using the rule set to perform enumeration verification on all the matching entities in the entity bucket to obtain conflicting matching entity pairs corresponding to the entity bucket, including: Allocating a thread block for each double-star mode rule in the rule set in the GPU, and the thread block is used to execute the enumeration task corresponding to the double-star mode rule; For any thread block, in the form of single instruction multiple data, pairwise verification is performed on all the matching entities in the entity bucket to detect whether the dependencies specified by the corresponding double-star mode rule are satisfied; If the dependencies specified by the corresponding double-star mode rule are not satisfied, determine the corresponding entity pair as a conflicting matching entity pair.

[0049] Among them, the embodiments of the present application make full use of the single instruction multiple data characteristics of the GPU. In the double-star matching enumeration stage, according to the task ID and the single-star beam entity ID, the single-star beam entities are evenly bound to different GPU threads. In the parallel mode of single instruction multiple data, the system can enumerate thousands of single-star beam entity pairs at the same time. Since there are few branch judgments in this process, the powerful parallel ability of the GPU can be fully utilized.

[0050] In addition, embodiments of the present application assign different double-star matching enumeration tasks to different thread blocks of the GPU, forming dependency-free task parallelism. Since the double-star matching enumeration tasks are independent of each other, the present application can adaptively submit enumeration tasks to the GPU. For example, if the current amount of enumeration tasks is small and the GPU load is not saturated, the present invention will submit other enumeration tasks to the idle thread blocks to fully utilize the GPU resources.

[0051] Optionally, the CPU is used to execute the single-star beam matching task. After bucketing all the matching entities according to the materialization attributes to obtain N entity buckets, the following steps are further included: Place the matching entities in the N entity buckets into the buffer of the CPU. If the buffer manager of the buffer detects that the matching entities exceed the memory capacity of the buffer, the excess part is swapped into the swap area of the storage area storing the target graph data. Submit the matching entities of each entity bucket in the buffer to the GPU through the GPU proxy.

[0052] Among them, as Figure 6 shown in the stand-alone graph data cleaning system, through the cooperation of buffer management and GPU proxy, the exchange of entities and results is realized, improving the efficient interaction between the CPU and the GPU.

[0053] Optionally, after traversing all the entity buckets to obtain all the conflicting matching entity pairs, the following steps are further included: Deposit all the conflicting matching entities into the CPU through the GPU proxy. The conflict repair of the conflicting matching entity pairs to obtain the graph data cleaning result includes: Use a result aggregator to perform conflict repair on the conflicting matching entities in the CPU to obtain the graph data cleaning result. Save the graph data cleaning result to the storage area storing the target graph data.

[0054] Among them, as Figure 6 shown in the stand-alone graph data cleaning system, through the GPU proxy, the conflicting matching entities are stored in the CPU, and the result aggregator in the CPU repairs the conflicting matching entities to obtain the data cleaning result.

[0055] Optionally, the obtaining of the target graph data and the rule set composed of the double-star pattern rules obtained by rule discovery based on the target graph data includes: The CPU obtains the target graph data from the storage area storing the target graph data. Perform rule discovery on the target graph data to obtain at least one double-star pattern rule, and form a rule set with all the double-star pattern rules.

[0056] Among them, under the architecture of the stand-alone graph data cleaning system as shown in Figure 6 , the target graph data is stored in the SSD disk, and correspondingly, the rule set is also stored in the SSD disk.

[0057] In the embodiment of the present application, the target graph data and the rule set composed of the double-star pattern rules obtained by rule discovery based on the target graph data are obtained. All the double-star pattern rules in the rule set are disassembled into single-star components, the component structure of each single-star component is determined, the single-star components with the same nodes and edges in the component structure are subjected to intersection operation and union operation to obtain single-star bundles including common sub-structures and complete structures. Based on each single-star bundle, the corresponding matching entities are matched from the target graph data, and all the matching entities are enumerated and verified using the rule set to obtain conflicting matching entity pairs, and the conflicting matching entity pairs are repaired to obtain the graph data cleaning result. By disassembling the double-star pattern rules and constructing single-star bundles for the disassembled single-star components, and using the single-star bundles to execute the matching entities, there is no need to repeatedly calculate the shared structure, effectively avoiding redundant calculations in the pattern matching stage and improving the execution efficiency of the system.

[0058] Refer to Figure 7 , which is a schematic flowchart of an optimized control method for stand-alone graph data cleaning provided in Embodiment 3 of the present application. As shown in Figure 7 , the memory segment can be evenly divided according to the number of buckets for bucketing the matching entities obtained from the preset graph data by the entity bucketer in the above step S202 to obtain memory slots corresponding to the number of buckets, which may include the following steps: Step S701, monitor the GPU task queue to obtain the real-time GPU task queue length.

[0059] Step S702, when the real-time GPU task queue length is less than or equal to the first threshold, enable the CPU to combine all single-star components that meet the combination requirements when performing the intersection operation and union operation on the single-star components with the same nodes and edges in the component structure to obtain single-star bundles including common sub-structures and complete structures.

[0060] Step S703, if the real-time GPU task queue length is less than or equal to the second threshold and greater than the first threshold, enable the CPU to combine all single-star components that meet the combination requirements and have a common sub-structure larger than the preset structure size when performing the intersection operation and union operation on the single-star components with the same nodes and edges in the component structure to obtain single-star bundles including common sub-structures and complete structures.

[0061] Step S704, when the length of the real-time GPU task queue is greater than the second threshold, prohibit the CPU from performing the intersection operation and union operation of single-star components whose component structures include the same nodes and edges, obtaining a single-star bundle including a common sub-structure and a complete structure, and directly output the to-be-processed single-star component as the single-star bundle.

[0062] In this embodiment, there are different graph data cleaning workloads in the actual scenario, such as CPU-intensive workloads and GPU-intensive workloads. If the computing loads of the CPU and GPU cannot be effectively balanced, the pipeline parallel model will degenerate into a sequential model. Specifically, when more rules are combined to form a single-star bundle, less repetitive matching work needs to be done on the CPU side; however, since the set of single-star bundle instances is a superset of the set of all single-star component instances, enumerating single-star bundle instances will introduce a greater computational overhead on the GPU side. Therefore, rule bundling can be regarded as a means of shifting the CPU workload to the GPU.

[0063] Among them, a GPU task queue monitor is set, and three rule combination modes are set according to different task loads.

[0064] 1. Greedy mode: When the length of the GPU task queue is less than or equal to the first threshold π1, the monitor considers that the current workload is CPU-intensive, so the system enters the greedy mode. In this mode, the system will combine all combinable rules to shift the workload on the CPU side to the GPU.

[0065] 2. Balanced mode: When the length of the GPU task queue is less than or equal to the second threshold π2 but greater than the first threshold π1, the monitor considers that the current workloads of the CPU and GPU are relatively balanced, so it will preferentially combine rules with larger shared sub-structures.

[0066] 3. Disabled mode: When the length of the GPU task queue is greater than the second threshold π2, the monitor considers that the current workload is GPU-intensive, so it will disable the rule combination optimization to avoid shifting the CPU load to the GPU.

[0067] For this application, for a large graph with 160 million nodes and 750 million edges in the Semantic Scholar dataset, it is superior to the latest system in both error detection and error repair tasks. For the latest baseline distributed system with a 32-node configuration, this application achieves at least an 8.09-fold performance improvement.

[0068] In the embodiment of this application, for different graph data cleaning workloads, the present invention adaptively adopts multiple rule combination modes by monitoring the GPU task queue to balance the computing loads of the CPU and GPU and improve the system performance.

[0069] An optimized control method for single-machine graph data cleaning corresponding to the above embodiments Figure 8 FIG. Figure 8 shows a structural block diagram of an optimized control device for single-machine graph data cleaning provided in Embodiment 4 of the present application. The above optimized control device for single-machine graph data cleaning can be applied to Figure 1 the server in

[0070] See Figure 8 , the optimized control device for single-machine graph data cleaning includes: A data acquisition module 81, configured to acquire target graph data and a rule set composed of double-star pattern rules obtained by rule discovery based on the target graph data, where the double-star pattern rules represent the association and dependency relationships between any two different entities; A pattern disassembling module 82, configured to disassemble all double-star pattern rules in the rule set into single-star components, and determine the component structure of each single-star component. The single-star component includes a predicate condition and a materialized attribute that need to be satisfied by one entity, and the component structure includes at least one node and an edge connecting two nodes; A pattern combining module 83, configured to perform intersection operation and union operation on single-star components with the same node and edge in the component structure to obtain a single-star bundle including a common sub-structure and a complete structure. The common sub-structure includes a shared node connection relationship and a shared predicate condition, and the complete structure includes all node connection relationships and all materialized attributes of all merged single-star components; A graph data cleaning module 84, configured to, based on each single-star bundle, match corresponding matching entities from the target graph data, use the rule set to enumerate and verify all matching entities, obtain conflicting matching entity pairs, and perform conflict repair on the conflicting matching entity pairs to obtain a graph data cleaning result.

[0071] Optionally, the optimized control device for single-machine graph data cleaning further includes: After matching corresponding matching entities from the target graph data based on each single-star bundle, perform bucketing on all matching entities according to the materialized attributes to obtain N entity buckets, where matching entities with the same materialized attribute are in the same entity bucket, and N is an integer greater than zero; The graph data cleaning module 84 includes: A conflicting entity pair determination unit, configured to, for any entity bucket, use the rule set to enumerate and verify all matching entities in the entity bucket to obtain conflicting matching entity pairs corresponding to the entity bucket; The entity bucket traversal unit is used to traverse all entity buckets to obtain all conflicting matching entity pairs.

[0072] Optionally, the GPU is used to execute the enumeration task. The conflicting entity pair determination unit includes: The thread block allocation subunit is used to allocate a thread block for each double-star pattern rule in the rule set in the GPU. The thread block is used to execute the enumeration task corresponding to the double-star pattern rule; The single instruction multiple data subunit is used to perform pairwise verification on all matching entities in the entity bucket in the form of single instruction multiple data for any thread block to detect whether the dependencies specified by the corresponding double-star pattern rule are satisfied; The conflicting entity pair determination subunit determines the corresponding entity pair as a conflicting matching entity pair if the dependencies specified by the corresponding double-star pattern rule are not satisfied.

[0073] Optionally, the CPU is used to execute the single-star beam matching task. The optimization control device for single-machine graph data cleaning further includes: The buffer management module is used to place the matching entities in the N entity buckets into the buffer of the CPU after bucketing all the matching entities according to the to-be-materialized attributes. If the buffer manager of the buffer detects that the matching entities exceed the memory capacity of the buffer, the excess part is swapped into the swap area of the storage area storing the target graph data; The GPU proxy module is used to submit the matching entities of each entity bucket in the buffer to the GPU through the GPU proxy.

[0074] Optionally, the optimization control device for single-machine graph data cleaning further includes: The conflicting entity feedback module is used to deposit all the conflicting matching entities into the CPU through the GPU proxy after traversing all the entity buckets to obtain all the conflicting matching entity pairs; The graph data cleaning module 84 includes: The graph data cleaning unit is used to use the result aggregator to perform conflict repair on the conflicting matching entities in the CPU to obtain the graph data cleaning result; The cleaning result storage unit is used to save the graph data cleaning result to the storage area storing the target graph data.

[0075] Optionally, the optimization control device for single-machine graph data cleaning further includes: The GPU monitoring module is used to monitor the GPU task queue to obtain the real-time length of the GPU task queue; The first combination control module is used to enable the CPU to combine all single-star components that meet the combination requirements when the length of the real-time GPU task queue is less than or equal to the first threshold, and when performing the intersection operation and union operation on single-star components with the same nodes and edges in the component structure to obtain a single-star bundle including a common sub-structure and a complete structure; The second combination control module is used to enable the CPU to combine all single-star components that meet the combination requirements and whose common sub-structure is larger than the preset structure size when the length of the real-time GPU task queue is less than or equal to the second threshold and greater than the first threshold, and when performing the intersection operation and union operation on single-star components with the same nodes and edges in the component structure to obtain a single-star bundle including a common sub-structure and a complete structure; The third combination control module is used to prohibit the CPU from performing the intersection operation and union operation on single-star components with the same nodes and edges in the component structure when the length of the real-time GPU task queue is greater than the second threshold, and directly output the single-star components to be processed as a single-star bundle.

[0076] Optionally, the data acquisition module 81 includes: A graph data acquisition unit for the CPU to acquire the target graph data from the storage area storing the target graph data; A rule discovery unit for performing rule discovery on the target graph data to obtain at least one double-star mode rule, and forming a rule set with all the double-star mode rules.

[0077] It should be noted that for the information interaction, execution process, etc. between the above modules, units, and sub-units, since they are based on the same concept as the method embodiment of the present application, their specific functions and the technical effects brought about can be specifically referred to in the method embodiment part, and will not be elaborated here.

[0078] Figure 9 This is a schematic structural diagram of a computer device provided in Embodiment 5 of the present application. As Figure 9 shown, the computer device in this embodiment includes: at least one processor ( Figure 9 only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, it implements the steps in any of the above-mentioned optimized control methods for single-machine graph data cleaning or the method embodiment of the optimized control method for single-machine graph data cleaning.

[0079] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand, Figure 9The examples of computer devices are merely illustrative and do not constitute a limitation on computer devices. A computer device may include more or fewer components than shown in the figures, or combine certain components, or have different components. For example, it may also include a network interface, a display screen, an input device, etc.

[0080] The so-called processor may be a CPU, and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0081] The memory includes a readable storage medium, an internal memory, etc. Among them, the internal memory may be the memory of the computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be the hard disk of the computer device, and in some other embodiments, it may also be an external storage device of the computer device. For example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the memory may also include both the internal storage unit of the computer device and the external storage device. The memory is used to store the operating system, application programs, a boot loader, data, and other programs, such as the program code of computer programs. The memory may also be used to temporarily store the data that has been output or will be output.

[0082] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be assigned to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above device can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0083] All or part of the processes in the above method embodiments of this application can also be completed by a computer program product. When the computer program product runs on a computer device, it enables the computer device to execute and implement the steps in the above method embodiments.

[0084] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0085] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0086] In the embodiments provided in this application, it should be understood that the disclosed device / computer device and method can be implemented in other ways. For example, the device / computer device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0087] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0088] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit it; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. An optimized control method for single-machine graph data cleaning, characterized in that including: obtaining a target graph data and a rule set composed of double-star pattern rules obtained by rule discovery based on the target graph data, where the double-star pattern rules characterize the association and dependency relationships between any two different entities; decomposing all the double-star pattern rules in the rule set into single-star components, determining the component structure of each single-star component, where the single-star component includes a predicate condition and a materialized property that an entity needs to satisfy, and the component structure includes at least one node and an edge connecting two nodes; performing intersection and union operations on single-star components with the same node and edge in the component structure to obtain single-star bundles including a common substructure and a complete structure, where the common substructure includes shared node connection relationships and shared predicate conditions, and the complete structure includes all node connection relationships and all materialized properties of all merged single-star components; based on each single-star bundle, matching corresponding matching entities from the target graph data, using the rule set to enumerate and verify all the matching entities, obtaining conflicting matching entity pairs, and repairing the conflicting matching entity pairs to obtain a graph data cleaning result.

2. The optimized control method for single-machine graph data cleaning according to claim 1, wherein After matching corresponding matching entities from the target graph data based on each single-star bundle, it further includes: bucketing all the matching entities according to the materialized properties to obtain N entity buckets, where matching entities with the same materialized property are in the same entity bucket, and N is an integer greater than zero; the using the rule set to enumerate and verify all the matching entities to obtain conflicting matching entity pairs includes: for any entity bucket, using the rule set to enumerate and verify all the matching entities in the entity bucket to obtain conflicting matching entity pairs corresponding to the entity bucket; traversing all entity buckets to obtain all conflicting matching entity pairs.

3. The optimized control method for single-machine graph data cleaning according to claim 2, characterized in that, using the GPU to execute the enumeration task, the using the rule set to enumerate and verify all the matching entities in the entity bucket to obtain conflicting matching entity pairs corresponding to the entity bucket includes: allocating a thread block for each double-star pattern rule in the rule set in the GPU, and the thread block is used to execute the enumeration task of the corresponding double-star pattern rule; for any thread block, in the form of single instruction multiple data, pairwise checking all the matching entities in the entity bucket to detect whether the dependencies specified by the corresponding double-star pattern rule are satisfied; if the dependencies specified by the corresponding double-star pattern rule are not satisfied, determining the corresponding entity pair as a conflicting matching entity pair.

4. The optimized control method for single-machine graph data cleaning according to claim 3, characterized in that using the CPU to execute the single-star bundle matching task, after bucketing all the matching entities according to the materialized properties to obtain N entity buckets, it further includes: placing the matching entities in the N entity buckets into the buffer of the CPU, and if the buffer manager of the buffer detects that the matching entities exceed the memory capacity of the buffer, replacing the excess part into the swap area of the storage area storing the target graph data; submitting the matching entities of each entity bucket in the buffer to the GPU through the GPU proxy.

5. The optimized control method for single-machine graph data cleaning according to claim 4, wherein After traversing all entity buckets to obtain all conflicting matching entity pairs, it further includes: Through the GPU proxy, store all conflicting matching entities in the CPU; The conflict repair of the conflicting matching entity pairs to obtain the graph data cleaning result includes: Use a result aggregator to perform conflict repair on the conflicting matching entities in the CPU to obtain the graph data cleaning result; Save the graph data cleaning result to the storage area storing the target graph data.

6. The optimized control method for single-machine graph data cleaning according to claim 4, wherein, It further includes: Monitor the GPU task queue to obtain the real-time GPU task queue length; When the real-time GPU task queue length is less than or equal to the first threshold, enable the CPU to perform the intersection operation and union operation of single-star components with the same nodes and edges in the component structure to obtain a single-star bundle including a common sub-structure and a complete structure, and combine all single-star components that meet the combination requirements; When the real-time GPU task queue length is less than or equal to the second threshold and greater than the first threshold, enable the CPU to perform the intersection operation and union operation of single-star components with the same nodes and edges in the component structure to obtain a single-star bundle including a common sub-structure and a complete structure, and combine all single-star components that meet the combination requirements and whose common sub-structure is larger than the preset structure size; When the real-time GPU task queue length is greater than the second threshold, prohibit the CPU from performing the intersection operation and union operation of single-star components with the same nodes and edges in the component structure to obtain a single-star bundle including a common sub-structure and a complete structure, and directly output the single-star components to be processed as a single-star bundle.

7. The optimized control method for single-machine graph data cleaning according to claim 4, characterized in that The obtaining of the target graph data and the rule set composed of the double-star pattern rules obtained by rule discovery based on the target graph data includes: The CPU obtains the target graph data from the storage area storing the target graph data; Perform rule discovery on the target graph data to obtain at least one double-star pattern rule, and form a rule set with all the double-star pattern rules.

8. An optimized control device for single-machine graph data cleaning, characterized in that It includes: A data acquisition module for obtaining the target graph data and the rule set composed of the double-star pattern rules obtained by rule discovery based on the target graph data, where the double-star pattern rules represent the association and dependency relationships between any two different entities; A pattern disassembling module for disassembling all the double-star pattern rules in the rule set into single-star components, determining the component structure of each single-star component, where the single-star component includes the predicate conditions and the to-be-materialized attributes that an entity needs to satisfy, and the component structure includes at least one node and an edge connecting two nodes; A pattern combination module for performing intersection operation and union operation on single-star components with the same nodes and edges in the component structure to obtain a single-star bundle including a common sub-structure and a complete structure, where the common sub-structure includes the shared node connection relationships and the shared predicate conditions, and the complete structure includes all the node connection relationships and all the to-be-materialized attributes of all the combined single-star components; A graph data cleaning module, which is used to match corresponding matching entities from the target graph data based on each single star beam, enumerate and verify all the matching entities using the rule set, obtain conflicting matching entity pairs, and repair the conflicts of the conflicting matching entity pairs to obtain the graph data cleaning result.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the optimized control method for single-machine graph data cleaning according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the optimized control method for single-machine graph data cleaning according to any one of claims 1 to 7.

Images