Automated entity splitting method, device, equipment and medium

The attribute splitting and derivation of entity tuples through logical rule groups solves the accuracy problem of automated entity splitting and improves data accuracy.

CN117708104BActive Publication Date: 2024-11-22SHENZHEN INST OF COMPUTING SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311563603.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-21
Publication Date
2024-11-22
Estimated Expiration
2043-11-21

AI Technical Summary

Technical Problem

The prior art is difficult to automatically and accurately split the wrongly merged entity tuples, making it difficult to guarantee data accuracy.

Method used

By obtaining the entity tuple to be split and the preset logical rule group, using the logical rule group to split the attributes in the entity tuple, perform attribute value derivation and conflict resolution, determine the final attribute value, and realize automated entity splitting .

Benefits of technology

It realizes automatic accurate splitting of entities, improving the use effect of data in scenarios with high accuracy requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117708104B_ABST
    Figure CN117708104B_ABST
Patent Text Reader

Abstract

The present application is applicable to the field of data processing technology, and in particular, relates to an automated entity splitting method, device, equipment and medium. The method obtains an entity tuple to be split and a preset logical rule group, uses the logical rule group to split the attributes of the same entity in the entity tuple into a split tuple, obtains at least one split tuple, uses the logical rule group to derive attribute values ​​for all attributes in each split tuple, obtains all attribute value results under each attribute in each split tuple, determines the final attribute values ​​of all attributes in each split tuple based on all attribute value results under each attribute in each split tuple, realizes automated entity splitting, and derives and verifies attribute values ​​after splitting, thereby obtaining a relatively accurate splitting result, which is helpful for using data in scenarios with high accuracy requirements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application is applicable to the field of data processing technology, and in particular relates to an automated entity splitting method, device, equipment and medium. Background Art

[0002] Currently, one of the most studied topics in data quality is entity resolution. The entity resolution problem is to identify tuples belonging to the same entity and merge these tuples into one tuple. Entity resolution is also known as record linkage, deduplication, merging / cleaning, and record matching. It has been a routine operation in many applications. Through machine learning or logical rules, a lot of research has been done on entity resolution.

[0003] A related problem involves splitting merged tuples composed of mismatched entities. In practice, different entities may be mistakenly merged into the same tuple. For example, in user-contributed projects, mismatched merged tuples caused by users collaborating on data can be observed. In third-party data or public data, the data has been processed by other parties. Since the third party's data processing method is not necessarily correct, different entities may also be mistakenly matched to the same tuple. In data lineage, data cleaning starts from the checkpoint. The mismatch of entity tuples may be caused by historical operations and is not easy to find. Entity splitting can effectively disassemble the wrong data, that is, entity splitting is the opposite of entity resolution. For example, role-based splitting is to split the attributes of the person by the person's role. However, in most applications, manual entity splitting can only be performed by moving attributes one by one. However, since the merge method used in the entity resolution process is unknown, it is impossible to find the key points of the wrong merge based on the merge method, and thus the entity cannot be accurately split.

[0004] Therefore, how to automatically split entities and obtain accurate split results in order to cope with scenarios with high data accuracy requirements has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the embodiments of the present application provide an automated entity splitting method, apparatus, device and medium to solve the problem of how to automatically split entities and obtain accurate splitting results, so as to cope with scenarios with high data accuracy requirements.

[0006] In a first aspect, an embodiment of the present application provides an automated entity splitting method, the automated entity splitting method comprising:

[0007] Obtaining an entity tuple to be split and a preset logical rule group, and using the logical rule group to split the attributes of the same entity in the entity tuple into a split tuple, to obtain at least one split tuple;

[0008] Using the logic rule group to deduce attribute values ​​for all attributes in each split tuple, to obtain all attribute value results for each attribute in each split tuple;

[0009] According to all the attribute value results under each attribute in each split tuple, the final attribute values ​​of all the attributes in each split tuple are determined.

[0010] In a second aspect, an embodiment of the present application provides an automated entity splitting device, the automated entity splitting device comprising:

[0011] A tuple splitting module is used to obtain an entity tuple to be split and a preset logical rule group, and use the logical rule group to split the attributes of the same entity in the entity tuple into a split tuple to obtain at least one split tuple;

[0012] An attribute derivation module, used to derive attribute values ​​of all attributes in each split tuple using the logic rule group, and obtain all attribute value results of each attribute in each split tuple;

[0013] The attribute determination module is used to determine the final attribute values ​​of all attributes in each split tuple according to all attribute value results under each attribute in each split tuple.

[0014] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the automated entity splitting method as described in the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the automated entity splitting method as described in the first aspect is implemented.

[0016] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application obtains an entity tuple to be split and a preset logical rule group, uses the logical rule group to split the attributes in the entity tuple that represent the same entity into a split tuple, and obtains at least one split tuple, uses the logical rule group to derive attribute values ​​for all attributes in each split tuple, and obtains all attribute value results for each attribute in each split tuple, determines the final attribute values ​​of all attributes in each split tuple based on all attribute value results for each attribute in each split tuple, realizes automated splitting of entities, and derives and verifies attribute values ​​after splitting, thereby obtaining relatively accurate splitting results, which is helpful for using data in scenarios with high accuracy requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 This is a schematic diagram of an application environment of an automated entity splitting method provided in Example 1 of the present application;

[0019] Figure 2 It is a flowchart of an automated entity splitting method provided in Example 2 of the present application;

[0020] Figure 3 It is a flowchart of an automated entity splitting method provided in Example 3 of the present application;

[0021] Figure 4 This is a flow chart of an automated entity splitting method provided in Embodiment 4 of the present application;

[0022] Figure 5 This is a flowchart of an automated entity splitting method provided in Example 5 of the present application;

[0023] Figure 6 This is a schematic diagram of a framework of an automated entity splitting method provided in Example 6 of the present application;

[0024] Figure 7 This is a flow chart of an automated entity splitting method provided in Embodiment 7 of the present application;

[0025] Figure 8 This is a flow chart of an automated entity splitting method provided in Embodiment 8 of the present application;

[0026] Fig. 9 This is a schematic diagram of the structure of an automated entity splitting device provided in Embodiment 9 of the present application;

[0027] Fig.10 It is a structural diagram of a computer device provided in Example 10 of the present application. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0030] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0032] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0034] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0035] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0036] The first embodiment of the present application provides an automated entity splitting method, which can be applied in the following cases: Figure 1 In an application environment, a server communicates with a client, and the server is used to carry the automated entity splitting method to provide data splitting services. The client can request data splitting services from the server by providing corresponding entity tuples, thereby obtaining split tuples. Of course, the server can also split the entity tuples stored by itself. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computer devices, personal digital assistants (PDAs), and other devices. The computer device corresponding to the server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0037] See also Figure 2 , is a flow chart of an automated entity splitting method provided in Embodiment 2 of the present application, wherein the automated entity splitting method is applied to Figure 1 In the server, the user corresponding to the client sends the entity tuple to be split to the server to trigger the server to perform the splitting task. Figure 2 As shown, the automated entity splitting method may include the following steps:

[0038] Step S201 , obtaining an entity tuple to be split and a preset logic rule group, and using the logic rule group to split the attributes of the same entity in the entity tuple into a split tuple, to obtain at least one split tuple.

[0039] In this embodiment, a tuple is a row or a column in a relational table in a database. If a row is represented as a tuple, the row of data corresponds to an entity. Each column in the row of data represents the attributes of the entity. The entity tuple is a row of data in the relational table.

[0040] In the process of collecting and organizing data, the attributes of two different entities may be mistakenly merged into one entity. For example, there are two entities named A. The gender attribute of the first entity A is "male (that is, the attribute value of the gender attribute)", and the age attribute of the second entity A is "25 years old (the attribute value of the age attribute)". After the two entities are mistakenly merged, the entity A becomes the gender attribute "male" and the age attribute "25 years old". Obviously, this is an incorrect fusion result.

[0041] It can be understood that the attribute values ​​of the attributes in the erroneously merged entity tuple represent more than one object. An object may refer to an object, item, person, place, etc. in the real world. For example, for an entity tuple, the corresponding entity is represented by a name. The entity name is Name1, which corresponds to multiple objects in reality. The corresponding gender attribute in its entity tuple is "male", and the corresponding medical attribute is "gynecology". Based on the principle of conflict, the gender attribute and the medical attribute are conflicting attributes. Based on this, the entity name Name1 can be divided into two split entities, such as the first split entity and the second split entity, wherein the name of the first split entity is Name1, the gender attribute is "male", and the medical attribute is empty, and the name of the second split entity is also Name1, the gender attribute is empty, and the medical attribute is "gynecology".

[0042] After splitting, for the same entity, its attributes should be classified into a split tuple, and the split tuple can only include the attributes split from the entity tuple. For example, if the entity tuple contains attribute 1, attribute 2, and attribute 3, attribute 1 and attribute 2 are split into a split tuple 1, and attribute 3 is split into an attribute tuple 2. Of course, the attributes in the split tuple and the attributes in the entity tuple can also be set to be equivalent. For attributes without attribute values, they can be left blank, that is, if the entity tuple contains attribute 1, attribute 2, and attribute 3, the split tuple should also contain attribute 1, attribute 2, and attribute 3.

[0043] The preset logic rule group includes at least one logic rule, wherein a logic rule φ is composed of a rule condition X and a result Y. If X is established, then Y is considered to be established. In other words, φ indicates that Y is a result obtained by logical deduction based on condition X. The logic rules in the preset logic rule group can be obtained offline, and of course, they can also be real-time logic rules before executing the method of this embodiment.

[0044] Logical rules can be used to determine whether the entity tuple contains the above-mentioned incorrect fusion, so as to determine whether the entity tuple needs to be split. For incorrectly fused entities, they need to be split. In addition, logical rules can also be used to assign values ​​to the attributes of the entity tuple, that is, logical rules can be used to match specific attributes to possible attribute values, so as to assign values ​​using the attribute values. In a database, a logical rule φ may be applied to different data, and h represents an application of the logical rule φ.

[0045] The logic rules in this embodiment may be obtained by manual design or by an automated rule discovery method. Specifically, the conditions X and results Y of the logic rules may be listed by enumeration by manual or automated discovery methods.

[0046] Step S202: Use the logic rule group to deduce attribute values ​​for all attributes in each split tuple to obtain all attribute value results for each attribute in each split tuple.

[0047] In this embodiment, for the split tuples obtained by the above splitting, the attributes therein may correspond to incorrect attribute values ​​or be empty. In order to ensure the correctness and validity of the split tuples, it is necessary to deduce the attributes in the split tuples to obtain accurate attribute values.

[0048] For any attribute in any split tuple, use the logical rule group to deduce the attribute value of the attribute. If the logical rule group includes more than one logical rule, it is necessary to use each logical rule in an iterative manner to deduce the attribute value of the attribute separately. Each logical rule corresponds to an attribute value result after the attribute value derivation, and thus there is at least one attribute value result under the attribute.

[0049] The attribute value derivation process is essentially a process of repairing the attribute value of the attribute, that is, applying logical rules to repair it. The repair result is the above-mentioned attribute value result. These attribute value results can be stored in a set U. Each attribute value result can be represented in the form of (t[A], c), where (t[A], c) indicates that the attribute value of attribute A of a split tuple t is c.

[0050] Using a logical rule φ for deduction, it can be characterized as:

[0051]

[0052] Among them, h is the application of the logic rule φ, U is the set of attribute value results under the attribute before the logic rule φ is derived, In order to add a new set of attribute value results, the application of the logic rule φ needs to meet the following conditions: φ is a valid logic rule, and There exists a new attribute value result derived by h, that is, the application of φ in A new (t[A], c) is added, where a valid logic rule may refer to a condition X in the logic rule that has passed the verification of the verification data. The logic rule verified by the verification data is deduced. As long as the logic rule and the verification data are correct, the deduction result is also correct.

[0053] Step S203, determining the final attribute values ​​of all attributes in each split tuple according to all attribute value results under each attribute in each split tuple.

[0054] In this embodiment, since there may be multiple attribute value results obtained through the above derivation, and for an attribute in a split tuple, the attribute value needs to be uniquely determined, it is necessary to analyze the attribute value results obtained by the above derivation to obtain the final attribute value of each attribute.

[0055] That is, if there are two attribute value results for an attribute, and there is a conflict between the two attribute value results, then a corresponding conflict resolution method needs to be used to obtain the exact attribute value of the attribute.

[0056] At this point, it should be considered that all attribute value sets of t[A] can be derived through the above step S202, which are recorded as attribute value set C, and the final attribute value of t[A] needs to be determined based on the attribute value set C. Among them, all attribute values ​​in the attribute value set C are screened to obtain the final attribute value, and the screening can be based on the calculation and comparison of the correlation of data, for example, the correlation between the attribute values ​​in the attribute value set C and other attributes of the split tuple t is calculated.

[0057] The embodiment of the present application obtains an entity tuple to be split and a preset logical rule group, uses the logical rule group to split the attributes in the entity tuple that represent the same entity into a split tuple, and obtains at least one split tuple, uses the logical rule group to derive attribute values ​​for all attributes in each split tuple, and obtains all attribute value results for each attribute in each split tuple, and determines the final attribute values ​​of all attributes in each split tuple based on all attribute value results for each attribute in each split tuple, thereby realizing automated splitting of the entity, and deriving and verifying the attribute values ​​after the splitting, thereby obtaining a more accurate splitting result, which is helpful for using data in scenarios with higher accuracy requirements.

[0058] See also Figure 3 , is a flow chart of an automated entity splitting method provided in Example 3 of the present application. Figure 3 As shown, in step S201, the attributes of the entity tuple that represent the same entity are split into a split tuple using a logical rule group to obtain at least one split tuple, and the following steps may also be included:

[0059] Step S301 : using a logical rule group, determining whether the attribute value of a first attribute and the attribute value of a second attribute in an entity tuple represent the same entity.

[0060] Step S302: If the attribute value of the first attribute and the attribute value of the second attribute represent the same entity, it is determined that the attribute value of the first attribute and the attribute value of the second attribute both belong to a split tuple.

[0061] Step S303: If the attribute value of the first attribute and the attribute value of the second attribute represent different entities, it is determined that the attribute value of the first attribute belongs to one split tuple, and the attribute value of the second attribute belongs to another split tuple.

[0062] In this embodiment, the first attribute and the second attribute are not the same attribute, and both are arbitrary attributes in the entity tuple, that is, to determine whether the attributes in the entity tuple represent the same entity, it is necessary to determine whether the attribute values ​​of the two attributes should refer to the same entity. When the attribute values ​​of the two attributes are determined by the logical rules, it is determined that the two attributes do not satisfy the logical rules, and then the two attributes are determined to represent different entities. For example, the above-mentioned gender attribute is "male" and the medical attribute is "gynecology". The attribute values ​​between the gender attribute and the medical attribute are determined to be not the same entity. The specific corresponding logical rules are: male corresponds to andrology, and female corresponds to gynecology. Therefore, the male and gynecology in the entity tuple do not belong to the same entity.

[0063] If two attributes represent the same entity, the attribute values ​​of the two attributes are split into one split tuple. If the two attributes represent different entities, the attribute value of one attribute is split into one split tuple, and the attribute value of the other attribute is split into another split tuple. At this time, the split tuple can include the attributes split into it, and can also include other attributes in the entity tuple, but there is no attribute value in the other attributes, which can be obtained through subsequent deduction.

[0064] See also Figure 4 , is a flow chart of an automated entity splitting method provided in Example 4 of the present application. Figure 4 As shown, the above step S203 determines the final attribute values ​​of all attributes in each split tuple according to all attribute value results under each attribute in each split tuple, and may further include the following steps:

[0065] Step S401: for any split tuple, the attributes in the split tuple with attribute value conflicts are taken as attributes to be resolved.

[0066] In this embodiment, in the process of performing attribute value derivation in step S202, for an attribute, if a logic rule is used to derive the attribute value of the attribute to obtain a first attribute value result, and another logic rule is used to derive the attribute value of the attribute to obtain a second attribute value result, whether there is a conflict between the first attribute value result and the second attribute value result, if there is a conflict, it means that the attribute is a to-be-resolved attribute with a conflict to be resolved. Of course, only when there is no conflict between all the attribute value results in the attribute, there is no need to resolve the conflict for the attribute.

[0067] For a logical rule, after deducing the attribute value of an attribute, two attribute value results may be obtained. You can also refer to the above process and take the attribute as the research object. As long as there is a conflict in the attribute value results under the attribute, it is an attribute to be resolved.

[0068] The above-mentioned conflict may refer to an essential difference or a large difference between two attribute values. The method for determining the conflict may be set according to demand. For example, if the two attribute values ​​are numerical, they are considered to be different or the difference between them is greater than a threshold, and they are determined to be a conflict. If the two attribute values ​​are character types, it is necessary to determine the character similarity or semantic similarity, and determine whether there is a conflict based on the similarity. For example, "andrology" and "gynecology" are two attribute values ​​with essential differences in semantics, and it can be determined that the two are in conflict.

[0069] Step S402, obtaining a verified attribute set of the split tuple.

[0070] In this embodiment, the verified attribute set includes verified attributes, and the corresponding attribute values ​​of the verified attributes are confirmed to belong to the corresponding split entity. For example, for split entity 1, it includes attribute 1, attribute 2 and attribute 3, among which the attribute values ​​corresponding to attribute 1 and attribute 2 are confirmed to belong to the split entity 1, then attribute 1 and attribute 2 are the verified attributes of the split tuple.

[0071] The verified attributes can be some attributes and corresponding attribute values ​​set according to requirements. Of course, since the entity tuple is the data obtained through entity parsing, although there are fusion errors, its attribute values ​​may be correct. Therefore, the attributes split from the entity tuple can also be used as the verified attributes in the split tuple. In addition, after the attributes of the split tuple are subsequently deduced, the derived attributes without conflict can also be used as the verified attributes in the split tuple.

[0072] Step S403: Screen all attribute value results in the attribute to be resolved according to the attribute values ​​of all attributes in the verified attribute set, and obtain the screened attribute value results as the final attribute value of the attribute to be resolved.

[0073] In this embodiment, the attributes in the verified attribute set can indicate partial features of the entity corresponding to the split tuple to a certain extent. Therefore, the attribute value of the attribute to be solved should satisfy the correlation with the partial feature. Therefore, based on this, all attribute value results of the attribute to be solved can be screened to obtain the relevant attribute value as the final attribute value.

[0074] For a split tuple, when using the preset logical rule group to derive the attribute value of each attribute, since the preset logical rule group needs to derive each attribute one by one, if the currently derived attribute is the first derived attribute in the split tuple, and the first derived attribute is the attribute to be resolved, then the preset verified attribute or the attribute split from the entity tuple can be used to filter all the attribute value results in the attribute to be resolved to obtain the final attribute value.

[0075] If the currently derived attribute is not the first derived attribute in the split tuple, and the attribute is the attribute to be resolved, you can use the preset verified attributes, the attributes split from the entity tuple, or the attributes with the final attribute value determined to filter all the attribute value results in the attribute to be resolved to obtain the final attribute value.

[0076] In an embodiment of the present application, the attribute value of the verified attribute can be used to screen the conflicting attribute value results to solve the attribute value conflict problem under the attribute, thereby determining the final attribute value of the attribute, and ensuring the uniqueness of the split tuple.

[0077] See also Figure 5 , is a flowchart of an automated entity splitting method provided in Example 5 of the present application, such as Figure 5 As shown, the step S402 of obtaining the verified attribute set of the split tuple may include the following steps:

[0078] Step S501: The attributes in the split tuple that do not have attribute value result conflicts and the unresolved attributes in the split tuple that have determined final attribute values ​​are taken as verified attributes.

[0079] Step S502: Obtain a verified attribute set based on the verified attributes.

[0080] In this embodiment, for the split tuple, after performing attribute value derivation, if there is no conflict in the attribute value result corresponding to the attribute, it means that the attribute is not a conflicting attribute, and the attribute value result corresponding to the attribute is its final attribute value, which can be used as the verified attribute of the split tuple. In addition, if the attribute is a conflicting attribute, but after conflict resolution (i.e., the above step S403), the final attribute value is obtained, which can also be used as the verified attribute of the split tuple.

[0081] In the embodiment of the present application, a verified attribute set is constructed based on the verified attributes. The verified attribute set can be continuously updated in the subsequent process of deducing the attributes of the split tuple, so that the attributes in the set are richer, which helps to improve the accuracy of conflict attribute resolution.

[0082] See also Figure 6 , is a flow chart of an automated entity splitting method provided in Example 6 of the present application, such as Figure 6 As shown, the above step S403, according to the attribute values ​​of all attributes in the verified attribute set, screens all attribute value results in the attribute to be resolved, and obtains the screened attribute value result as the final attribute value of the attribute to be resolved, which may include the following steps:

[0083] Step S601, using the pre-trained association analysis model, calculate the association strength between each attribute value result in the attribute to be resolved and the attribute values ​​of all attributes in the verified attribute set, and obtain the association strength score corresponding to each attribute value result in the attribute to be resolved.

[0084] In this embodiment, the pre-trained association analysis model can be a model based on a neural network architecture, which can perform correlation analysis on input data and target data and output a correlation strength score, wherein the input data is an attribute value result in the attribute to be solved, and the target data is the attribute value of all attributes in the verified attribute set.

[0085] Step S602: Filter the association strength scores of all attribute value results in the attribute to be resolved and obtain the attribute value result with the highest association strength score as the target result, and determine the target result as the final attribute value of the attribute to be resolved.

[0086] In this embodiment, after calculation in the above step S601, the association strength score corresponding to each attribute value result in the attribute to be resolved is obtained, and the highest association strength score is determined from all the association strength scores. The highest association strength score is used as the target result, which indicates that the corresponding attribute value has the strongest correlation with the attributes in the verified attribute set. Therefore, the reliability that the target result is the correct attribute value is the highest, that is, the target result is used as the final attribute value of the attribute to be resolved.

[0087] For example, a conflict is found in the attribute value results of splitting tuple t[A], and the conflict is resolved by the following strategy based on attribute relevance:

[0088] Assume that the set of all attribute values ​​of t[A] that can be deduced through logical rules is C, and assume that the set of all verified attributes of t is By fine-tuning a large-scale language model (such as Bert), a model M for judging attribute relevance can be trained. Specifically, given a candidate attribute value result c in C, The candidate attribute value result c and all verified attributes will be output The association strength of t[A] is taken, and the value with the highest association in set C is taken as the final attribute value of t[A].

[0089] See also Figure 7 , is a flow chart of an automated entity splitting method provided in Example 7 of the present application. Figure 7 As shown, the logical rule group includes at least two logical rules. In step S202, the logical rule group is used to deduce attribute values ​​of all attributes in each split tuple to obtain all attribute value results under each attribute in each split tuple, which may include the following steps:

[0090] Step S701, obtaining verification data, and verifying each logic rule according to the verification data to obtain the verified logic rules.

[0091] In this embodiment, the logical rule group contains more than two logical rules. Before using each logical rule to pursue and execute each attribute, it is necessary to verify whether the logical rule is valid. At this time, verification data is needed. The verification data can be pre-set data for verifying condition X in the logical rule. The derivation results after each derivation can also be collected in the subsequent derivation process to update the verification data.

[0092] Step S702: for any split tuple, any attribute in the split tuple is used as the current derivation attribute, and each verified logical rule is called in turn to derive the attribute value of the current derivation attribute to obtain all attribute value results under the current derivation attribute.

[0093] In this embodiment, each logical rule is used for pursuit execution to obtain a pursuit sequence for the current derived attribute, that is, the logical rules are {φ1, φ2, ..., φn}, and the corresponding logical rules are executed from 1 to n in sequence, and finally the attribute value result set of the current derived attribute is obtained.

[0094] Step S703, traverse all attributes in the split tuple and all split tuples to obtain all attribute value results under each attribute in each split tuple.

[0095] In this embodiment, for a split tuple, each attribute needs to be processed by the above step S702 to obtain an attribute value result set, so the chasing result of the split tuple is from the derivation of the first attribute to the derivation of the last attribute, and a chasing sequence is obtained, as follows:

[0096]

[0097] in, is the result set of attribute values ​​derived for the first attribute, The resulting set of attribute values ​​for the last attribute derivation. The pursuit process ends when no logic rules can be applied anymore.

[0098] See also Figure 8 , is a flow chart of an automated entity splitting method provided in Example 8 of the present application. Figure 8 As shown, the above step S702 of performing attribute value derivation on the current derived attribute to obtain all attribute value results under the current derived attribute may include the following steps:

[0099] Step S801: Use the first logical rule to derive the attribute value of the current derivation attribute to obtain a first attribute value result.

[0100] Step S802: Use the second logic rule to derive the attribute value of the current derived attribute to obtain a second attribute value result.

[0101] Step S803: Use a preset data structure to store the attribute value results after attribute value deduction for each verified logical rule.

[0102] In this embodiment, the first logic rule is any logic rule among the verified logic rules, and the second logic rule is a logic rule among the verified logic rules after the first logic rule is executed.

[0103] Among them, if it is detected that the first attribute value result is the same as the second attribute value result, one of the first attribute value result and the second attribute value result is retained as the attribute value result of the current derived attribute; if it is detected that the first attribute value result and the second attribute value result are not the same, the first attribute value result and the second attribute value result are retained as the attribute value results of the current derived attribute; that is, after using a logical rule to derive the attribute value, it is necessary to compare with the attribute value result that has been derived before to determine whether it is the same as the attribute value result that has been derived before. If it is the same, only one attribute value result needs to be retained; if it is not the same, both attribute value results need to be retained. In the subsequent determination of whether it is a conflicting attribute to be resolved, if an attribute has two or more attribute value results, it is a conflicting attribute to be resolved; if an attribute has only one attribute value result, it is not a conflicting attribute.

[0104] If it is detected that the use of the second logic rule depends on the attribute value result of the first logic rule, the first attribute value result and the second logic rule in the preset data structure are called to derive the attribute value of the current derived attribute to obtain the second attribute value result. Since it is difficult to enumerate the application of logic rules, and the application of a logic rule may depend on the results of the application of other logic rules used for attribute value derivation before, a special data structure is designed to record the attribute value results that can only be temporarily stored in the application of logic rules, so that based on the attribute value results stored in the data structure, the application of logic rules can be effectively enumerated and reused.

[0105] In addition, for the verification of logical rules, there is no need to check again the application of logical rules that have been used without exception. Based on the attribute value results and corresponding logical rules stored in the data structure, only the application of affected or unverified logical rules can be checked.

[0106] This embodiment is applied in the data set of the real Internet Movie Database (IMDb) for verification, and the overall splitting accuracy can reach 92%. In addition, when the number of tuples in the data set reaches 1,057,217, the running time of the method of the present application is only 1481 seconds. When the scaling factor of the data set changes from 20% to 100%, the preset data structure is maintained to record the temporary attribute value results, and the application of logical rules can be effectively enumerated and reused, which improves the efficiency of the operation. Compared with the method of simply enumerating all application results, the running speed is 13.5 times faster. When more logical rules are used to perform derivation, the method of the present application requires more time. Even so, compared with the method of simply enumerating all application results, the running speed is still 11.8 times.

[0107] Corresponding to the automated entity splitting method of the above embodiment, Fig. 9 The structure block diagram of the automatic entity splitting device provided in the ninth embodiment of the present application is shown. The automatic entity splitting device is applied to Figure 1 In the server, the user corresponding to the client sends the entity tuple to be split to the server to trigger the server to perform the splitting task. For ease of explanation, only the part related to the embodiment of the present application is shown.

[0108] See also Fig. 9 , the automated entity splitting device comprises:

[0109] The tuple splitting module 91 is used to obtain an entity tuple to be split and a preset logic rule group, and use the logic rule group to split the attributes of the same entity in the entity tuple into a split tuple to obtain at least one split tuple;

[0110] The attribute derivation module 92 is used to derive attribute values ​​of all attributes in each split tuple using a logical rule group to obtain all attribute value results of each attribute in each split tuple;

[0111] The attribute determination module 93 is used to determine the final attribute values ​​of all attributes in each split tuple according to all attribute value results under each attribute in each split tuple.

[0112] Optionally, the tuple splitting module 91 includes:

[0113] an attribute value determination unit, used to determine whether an attribute value of a first attribute and an attribute value of a second attribute in the entity tuple represent the same entity using a logical rule group, wherein the first attribute and the second attribute are not the same attributes and are both arbitrary attributes in the entity tuple;

[0114] A first splitting unit, configured to determine that the attribute value of the first attribute and the attribute value of the second attribute both belong to a split tuple if the attribute value of the first attribute and the attribute value of the second attribute represent the same entity;

[0115] The second splitting unit is used to determine that the attribute value of the first attribute belongs to one split tuple and the attribute value of the second attribute belongs to another split tuple if the attribute value of the first attribute and the attribute value of the second attribute represent different entities.

[0116] Optionally, the attribute determination module 93 includes:

[0117] A conflict attribute determination unit, for determining, for any split tuple, an attribute in the split tuple with conflicting attribute values ​​as an attribute to be resolved;

[0118] An attribute set determination unit, used to obtain a verified attribute set of the split tuple, wherein the verified attribute set includes attributes that have passed verification;

[0119] The final attribute value determination unit is used to screen all attribute value results in the attribute to be resolved according to the attribute values ​​of all attributes in the verified attribute set, and obtain the screened attribute value result as the final attribute value of the attribute to be resolved.

[0120] Optionally, the verified attribute determination unit comprises:

[0121] The attribute determination subunit is used to treat the attributes in the split tuple that do not have attribute value result conflicts and the unresolved attributes in the split tuple whose final attribute values ​​have been determined as verified attributes;

[0122] The attribute set determination subunit is used to obtain a verified attribute set according to the attributes that have passed verification.

[0123] Optionally, the final attribute value determination unit includes:

[0124] The association calculation subunit is used to use the pre-trained association analysis model to calculate the association strength between each attribute value result in the attribute to be solved and the attribute values ​​of all attributes in the verified attribute set, and obtain the association strength score corresponding to each attribute value result in the attribute to be solved;

[0125] The attribute value determination subunit is used to select the attribute value result with the highest association strength score from the association strength scores of all attribute value results in the attribute to be solved as the target result, and determine the target result as the final attribute value of the attribute to be solved.

[0126] Optionally, the logic rule group includes at least two logic rules, and the attribute derivation module 92 includes:

[0127] A rule verification unit is used to obtain verification data, verify each logic rule according to the verification data, and obtain the verified logic rules;

[0128] The attribute derivation unit, for any split tuple, takes any attribute in the split tuple as the current derivation attribute, calls each verified logical rule in turn, derives the attribute value of the current derivation attribute, and obtains all attribute value results under the current derivation attribute;

[0129] The loop execution unit is used to traverse all attributes in the split tuple and all split tuples to obtain all attribute value results under each attribute in each split tuple.

[0130] Optionally, the attribute derivation unit includes:

[0131] A first attribute derivation subunit is used to derive an attribute value of a current derivation attribute using a first logic rule to obtain a first attribute value result, wherein the first logic rule is any logic rule among the logic rules that have passed verification;

[0132] A second attribute derivation subunit is used to derive an attribute value of a current derivation attribute using a second logic rule to obtain a second attribute value result, wherein the second logic rule is a logic rule after the first logic rule is executed among the verified logic rules;

[0133] The attribute value storage subunit is used to use a preset data structure to store the attribute value results after the attribute value is derived by each verified logical rule;

[0134] If it is detected that the first attribute value result is the same as the second attribute value result, one of the first attribute value result and the second attribute value result is retained as the attribute value result of the current attribute derivation; if it is detected that the first attribute value result and the second attribute value result are different, the first attribute value result and the second attribute value result are retained as the attribute value results of the current attribute derivation;

[0135] If it is detected that the use of the second logic rule depends on the attribute value result of the first logic rule, the first attribute value result and the second logic rule in the preset data structure are called to derive the attribute value of the current derivation attribute to obtain the second attribute value result.

[0136] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0137] Fig.10 This is a schematic diagram of the structure of a computer device provided in Example 10 of the present application. Fig.10As shown, the computer device of this embodiment includes: at least one processor ( Fig.10 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned embodiments of the automated entity splitting method are implemented.

[0138] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Fig.10 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0139] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0140] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments may also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory may also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of computer programs, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0141] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0142] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing the computer program product.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An automated entity splitting method, characterized in that: The automated entity splitting method comprises: Obtaining an entity tuple to be split and a preset logical rule group, and using the logical rule group to split the attributes of the same entity in the entity tuple into a split tuple, to obtain at least one split tuple; Using the logic rule group to deduce attribute values ​​for all attributes in each split tuple, to obtain all attribute value results for each attribute in each split tuple; Determine the final attribute values ​​of all attributes in each split tuple according to all attribute value results under each attribute in each split tuple; The logic rule group includes at least two logic rules, and the use of the logic rule group to derive attribute values ​​for all attributes in each split tuple to obtain all attribute value results for each attribute in each split tuple includes: Acquire verification data, and verify each logic rule according to the verification data to obtain the verified logic rules; For any split tuple, take any attribute in the split tuple as the current derivation attribute, call each verified logical rule in turn, derive the attribute value of the current derivation attribute, and obtain all attribute value results under the current derivation attribute; Traverse all attributes in the split tuple and all split tuples to obtain all attribute value results under each attribute in each split tuple; Determining the final attribute values ​​of all attributes in each split tuple according to all attribute value results under each attribute in each split tuple includes: For any split tuple, the attributes in the split tuple with attribute value conflicts are taken as attributes to be resolved; Acquire a verified attribute set of the split tuple, wherein the verified attribute set includes attributes that have passed verification; According to the attribute values ​​of all the attributes in the verified attribute set, all the attribute value results in the attribute to be resolved are screened, and the screened attribute value results are obtained as the final attribute value of the attribute to be resolved.

2. The automated entity splitting method according to claim 1, characterized in that: The step of using the logical rule group to split the attributes of the same entity in the entity tuple into a split tuple to obtain at least one split tuple includes: Using the logic rule group, determining whether an attribute value of a first attribute and an attribute value of a second attribute in the entity tuple represent the same entity, wherein the first attribute and the second attribute are not the same attribute and are both arbitrary attributes in the entity tuple; If the attribute value of the first attribute and the attribute value of the second attribute represent the same entity, determining that the attribute value of the first attribute and the attribute value of the second attribute both belong to a split tuple; If the attribute value of the first attribute and the attribute value of the second attribute represent different entities, it is determined that the attribute value of the first attribute belongs to one split tuple, and the attribute value of the second attribute belongs to another split tuple.

3. The automated entity splitting method according to claim 1, characterized in that: The step of obtaining a verified attribute set of the split tuple includes: The attributes in the split tuple that do not have attribute value result conflicts and the unresolved attributes in the split tuple whose final attribute values ​​have been determined are taken as verified attributes; A verified attribute set is obtained according to the verified attributes.

4. The automated entity splitting method according to claim 1, characterized in that: The step of screening all the attribute value results in the attribute to be resolved according to the attribute values ​​of all the attributes in the verified attribute set, and obtaining the screened attribute value results as the final attribute value of the attribute to be resolved, includes: Using the pre-trained association analysis model, calculate the association strength between each attribute value result in the attribute to be solved and the attribute values ​​of all attributes in the verified attribute set, and obtain the association strength score corresponding to each attribute value result in the attribute to be solved; The attribute value result with the highest association strength score is selected from the association strength scores of all attribute value results in the attribute to be resolved as the target result, and the target result is determined to be the final attribute value of the attribute to be resolved.

5. The automated entity splitting method according to claim 1, characterized in that: The performing attribute value derivation on the current derived attribute to obtain all attribute value results under the current derived attribute includes: Using a first logic rule to derive an attribute value of the current derived attribute to obtain a first attribute value result, wherein the first logic rule is any logic rule among the logic rules that have passed the verification; Using a second logic rule to derive an attribute value of the current derived attribute to obtain a second attribute value result, wherein the second logic rule is a logic rule after the first logic rule is executed among the verified logic rules; Use the preset data structure to store the attribute value results after the attribute value is derived by each verified logical rule; If it is detected that the first attribute value result is the same as the second attribute value result, one of the first attribute value result and the second attribute value result is retained as the attribute value result of the current derived attribute; if it is detected that the first attribute value result is different from the second attribute value result, the first attribute value result and the second attribute value result are retained as the attribute value results of the current derived attribute; If it is detected that the use of the second logical rule depends on the attribute value result of the first logical rule, the first attribute value result and the second logical rule in the preset data structure are called to derive the attribute value of the current derived attribute to obtain the second attribute value result.

6. An automated entity splitting device, characterized in that: The automated entity splitting device comprises: A tuple splitting module is used to obtain an entity tuple to be split and a preset logical rule group, and use the logical rule group to split the attributes of the same entity in the entity tuple into a split tuple to obtain at least one split tuple; An attribute derivation module, used to derive attribute values ​​of all attributes in each split tuple using the logic rule group, and obtain all attribute value results of each attribute in each split tuple; An attribute determination module, used for determining final attribute values ​​of all attributes in each split tuple according to all attribute value results under each attribute in each split tuple; The logic rule group includes at least two logic rules, and the attribute derivation module includes: A rule verification unit, used to obtain verification data, verify each logic rule according to the verification data, and obtain a verified logic rule; An attribute derivation unit is used to, for any split tuple, take any attribute in the split tuple as the current derivation attribute, call each verified logical rule in turn, derive the attribute value of the current derivation attribute, and obtain all attribute value results under the current derivation attribute; A loop execution unit, used for traversing all attributes in the split tuple and all split tuples, and obtaining all attribute value results under each attribute in each split tuple; The attribute determination module comprises: A conflicting attribute determination unit, configured to, for any split tuple, take the attribute in the split tuple that has a conflicting attribute value result as an attribute to be resolved; An attribute set determining unit, configured to obtain a verified attribute set of the split tuple, wherein the verified attribute set includes attributes that have passed verification; The final attribute value determination unit is used to screen all attribute value results in the attribute to be resolved according to the attribute values ​​of all attributes in the verified attribute set, and obtain the screened attribute value result as the final attribute value of the attribute to be resolved.

7. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the automated entity splitting method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the automated entity splitting method according to any one of claims 1 to 5 is implemented.