Method, device, equipment and medium for cleaning data of incorrectly matched entities

Abstract

The present application is applicable to the field of data cleaning technology, and in particular, relates to a method, device, equipment and medium for cleaning data of mismatched entities. The method performs anomaly detection on the acquired data tuple, obtains an entity tuple characterized as abnormal and an abnormal attribute in the entity tuple, detects the entity type of the entity tuple, and if it is detected that the entity type of the entity tuple is a mixed entity type, the entity tuple is determined to be a mixed entity, and according to each abnormal attribute and the corresponding attribute value in the mixed entity, a split entity corresponding to each abnormal attribute is constructed, and each split entity is respectively corrected for the abnormality to obtain a corrected split entity, and all corrected split entities are determined to be the result of cleaning the mixed entity, and the split entities are respectively corrected and cleaned by finding the mixed entity and constructing a split entity for each abnormal attribute in the mixed entity, so as to avoid data being discarded.

Description

Technical Field

[0001] The present application is applicable to the field of data cleaning technology, and in particular, relates to a method, device, equipment and medium for cleaning data of mismatched entities. Background Art

[0002] At present, as the sources of data increase, data usually have data quality problems, such as missing values, outliers and conflicting attributes. The reasons for the generation of missing values ​​and outliers may include mechanical problems (such as hardware failure) and privacy protection. Such data needs to be cleaned before it can be used downstream. In addition to the above reasons, the generation of conflicting attributes may also be caused by incorrect data fusion (i.e. entity resolution). For example, two people named Zhang San, one of whom is from Shenzhen and the other from Guangxi, the information of two people with the same name is mistakenly merged together. For conflicting attributes caused by incorrect data fusion, in the process of cleaning data, if one of the attributes (such as the city is Shenzhen) is assumed to be correct, and other attributes are corrected (such as the province is corrected from Guangxi to Guangdong), although the information of one entity is successfully corrected (such as Zhang San in Shenzhen), the information of another entity is also lost (i.e. another Zhang San in Guangxi). It can be seen that in the existing process of cleaning entities that are incorrectly matched together, information loss will occur, which will seriously affect the performance of downstream applications using this data.

[0003] Therefore, how to identify, split, and process mismatched entities to facilitate data cleaning while avoiding data loss has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a method, device, equipment and medium for cleaning data of mismatched entities to solve the problem of how to identify, split and process mismatched entities so as to facilitate data cleaning while avoiding data loss.

[0005] In a first aspect, an embodiment of the present application provides a method for cleaning data of mismatched entities, the cleaning method comprising:

[0006] Performing anomaly detection on the acquired data tuples to obtain entity tuples characterized as abnormal and abnormal attributes in the entity tuples, wherein the data tuples include at least one of the entity tuples;

[0007] Detecting the entity type of the entity tuple, and if it is detected that the entity type of the entity tuple is a mixed entity type, determining that the entity tuple is a mixed entity, wherein the entity tuple corresponding to the mixed entity type represents more than one object;

[0008] According to each abnormal attribute and the corresponding attribute value in the mixed entity, construct a segmentation entity corresponding to each abnormal attribute, wherein each segmentation entity includes an abnormal attribute and a corresponding attribute value;

[0009] Anomalies are corrected for each segmented entity to obtain a corrected segmented entity, and all corrected segmented entities are determined to be the result of cleaning the mixed entity.

[0010] In a second aspect, an embodiment of the present application provides a device for cleaning data of an incorrectly matched entity, the cleaning device comprising:

[0011] An anomaly detection module, used to perform anomaly detection on the acquired data tuples to obtain entity tuples characterized as abnormal and abnormal attributes in the entity tuples, wherein the data tuples include at least one of the entity tuples;

[0012] a type detection module, configured to detect the entity type of the entity tuple, and if it is detected that the entity type of the entity tuple is a mixed entity type, determine that the entity tuple is a mixed entity, wherein the entity tuple corresponding to the mixed entity type represents more than one object;

[0013] An entity segmentation module, used for constructing a segmentation entity corresponding to each abnormal attribute according to each abnormal attribute and the corresponding attribute value in the hybrid entity, wherein each segmentation entity includes an abnormal attribute and a corresponding attribute value;

[0014] The entity cleaning module is used to perform abnormal correction on each segmented entity to obtain a corrected segmented entity, and determine that all corrected segmented entities are the results of cleaning the mixed entity.

[0015] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cleaning method described in the first aspect when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cleaning method described in the first aspect is implemented.

[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application performs anomaly detection on the acquired data tuples to obtain entity tuples characterized as abnormal and abnormal attributes in the entity tuples, detects the entity type of the entity tuple, and if it is detected that the entity type of the entity tuple is a mixed entity type, determines that the entity tuple is a mixed entity, constructs a split entity corresponding to each abnormal attribute in the mixed entity and the corresponding attribute value, performs anomaly correction on each split entity to obtain a corrected split entity, determines that all corrected split entities are the results of cleaning the mixed entity, finds the mixed entity and constructs a split entity for each abnormal attribute in the mixed entity, so as to correct and clean the split entities respectively, thereby avoiding data discard. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 This is a schematic diagram of an application environment of a method for cleaning data of an incorrectly matched entity provided in the first embodiment of the present application;

[0020] Figure 2 It is a flowchart of a method for cleaning data of an incorrectly matched entity provided in the second embodiment of the present application;

[0021] Figure 3 It is a flowchart of a method for cleaning data of an incorrectly matched entity provided in Embodiment 3 of the present application;

[0022] Figure 4 It is a flowchart of a method for cleaning data of an incorrectly matched entity provided in the fourth embodiment of the present application;

[0023] Figure 5 It is a flowchart of a method for cleaning data of an incorrectly matched entity provided in Embodiment 5 of the present application;

[0024] Figure 6 It is a schematic diagram of a cleaning method provided in Example 5 of the present application;

[0025] Figure 7 It is a flowchart of a method for cleaning data of an incorrectly matched entity provided in Example 6 of the present application;

[0026] Figure 8It is a flowchart of a method for cleaning data of an incorrectly matched entity provided in Embodiment 7 of the present application;

[0027] Fig. 9 It is a structural schematic diagram of a device for cleaning data of an erroneously matched entity provided in Embodiment 8 of the present application;

[0028] Fig.10 It is a structural diagram of a computer device provided in Example 9 of the present application. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0030] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0031] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0032] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0033] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0034] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0035] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0036] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0037] A method for cleaning data of mismatched entities provided in the first embodiment of the present application can be applied in the following aspects: Figure 1 In an application environment, the server communicates with the client, and the server is used to carry the data cleaning method to provide data cleaning services. The client can request data cleaning services from the server by providing corresponding data tuples to obtain cleaned data. Of course, the server can also clean the data tuples stored by itself. Among them, the client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computer devices, personal digital assistants (PDAs), and other devices. The computer device corresponding to the server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0038] See also Figure 2 , is a flow chart of a method for cleaning data of an incorrectly matched entity provided in the second embodiment of the present application, wherein the method for cleaning data of an incorrectly matched entity is applied to Figure 1 In the server, the user corresponding to the client sends the data to be cleaned (i.e., data tuples) to the server to trigger the server to perform the cleaning task. Figure 2 As shown, the method for cleaning the data of the mismatched entity may include the following steps:

[0039] Step S201 , performing anomaly detection on the acquired data tuples to obtain entity tuples characterized as abnormal and abnormal attributes in the entity tuples.

[0040] In this embodiment, a tuple is a row or a column in a relational table in a database. If a row is represented as a tuple, the row of data corresponds to an entity. Each column in the row of data represents the attributes of the entity. The entity tuple is a row of data in the relational table. The data tuple includes at least one entity tuple. Specifically, the data tuple can be a relational table.

[0041] In the process of collecting and organizing data, the attributes of two different entities may be mistakenly merged into one entity. For example, there are two entities named A. The gender attribute of the first entity A is "male (that is, the attribute value of the gender attribute)", and the age attribute of the second entity A is "25 years old (the attribute value of the age attribute)". After the two entities are mistakenly merged, the entity A is obtained, with the gender attribute of "male" and the age attribute of "25 years old". Obviously, this is an incorrect fusion result and a manifestation of entity abnormality.

[0042] In addition, during the data collection and sorting process, there may be contradictory attribute values ​​in a collected entity. For example, entity B has a gender attribute of "male" and a medical attribute of "gynecology". Since men should not seek medical treatment in gynecology, there should be errors in the attributes of entity B, which is a manifestation of entity anomalies. Of course, entity anomalies also include anomalies such as empty attributes or exceeding the limit.

[0043] For data tuples, the anomaly detection method can be used to filter out abnormal entity tuples, and the abnormal attributes corresponding to the abnormal entity tuples can also be determined. Among them, the object of anomaly detection can be an attribute value. If the attribute value is abnormal, the corresponding attribute is also an abnormal attribute, and the corresponding entity tuple is the abnormal entity tuple.

[0044] By analyzing the characteristics of the attributes, we can get the corresponding detection rules. These detection rules can be used to automatically detect anomalies. The rules can be constructed based on logical conditions or machine learning models. Of course, other forms of rules can also be used to build rules. It is necessary to ensure that the constructed rules can automatically detect anomalies.

[0045] Step S202 , detecting the entity type of the entity tuple, and if it is detected that the entity type of the entity tuple is a mixed entity type, determining that the entity tuple is a mixed entity.

[0046] In this embodiment, for the above-mentioned abnormal entity tuples, there are single entities and mixed entities, wherein the mixed entity is a mixed entity type, and the entity tuple corresponding to the mixed entity type represents more than one object, and the single entity is a single entity type, and the entity tuple corresponding to the single entity type represents only one object.

[0047] An object may refer to an object, article, person, place, etc. in the real world. For example, for an entity tuple, the corresponding entity in the real world is city M, and the city M is a unique object, then the entity tuple is a single entity. For another example, for an entity tuple, the corresponding entity is represented by a name, and there are multiple person objects corresponding to the name in the real world, then the entity tuple is a mixed entity.

[0048] Step S203: construct a segmentation entity corresponding to each abnormal attribute according to each abnormal attribute and the corresponding attribute value in the mixed entity.

[0049] In this embodiment, when the hybrid entity is subjected to anomaly detection in the above step S201, all abnormal attributes in the hybrid entity are determined, wherein the characteristics of the abnormal attributes include conflict anomalies, empty set anomalies, error anomalies, etc. For the hybrid entity, since the hybrid entity corresponds to more than one object, the abnormal attribute characterized by the conflict anomaly should not belong to one object. Therefore, based on the abnormal attribute characterized by the conflict anomaly, the hybrid entity can be segmented into segmented entities representing different objects, that is, one segmented entity corresponds to one object.

[0050] Among them, each split entity contains an abnormal attribute and a corresponding attribute value, and the attributes in each split entity correspond one-to-one to the attributes of the mixed entity, that is, the mixed entity contains attribute 1, attribute 2 and attribute 3, then each split entity also includes attribute 1, attribute 2 and attribute 3. If attribute 2 and attribute 3 are abnormal attributes with conflicting anomalies, they can be divided into two split entities. The attribute value corresponding to attribute 2 in the mixed entity is used as the attribute value of attribute 2 of one split entity, and the attribute value corresponding to attribute 3 in the mixed entity is used as the attribute value of attribute 3 of another split entity. Attributes that do not have corresponding attribute values ​​in each split entity can be left blank, or a random or preset value can be written.

[0051] For example, for a mixed entity, the entity name is Name1, which corresponds to multiple objects in reality. The corresponding gender attribute in its entity tuple is "male", and the corresponding medical attribute is "gynecology". Based on the principle of conflict, the gender attribute and the medical attribute are conflicting attributes. Based on this, the entity name Name1 can be divided into two split entities, such as the first split entity and the second split entity, where the name of the first split entity is Name1, the gender attribute is "male", and the medical attribute is empty. The name of the second split entity is also Name1, the gender attribute is empty, and the medical attribute is "gynecology".

[0052] Step S204, performing abnormal correction on each segmented entity to obtain a corrected segmented entity, and determining that all corrected segmented entities are the result of cleaning the mixed entity.

[0053] In this embodiment, after entity segmentation in the above step S203, the obtained segmented entity is an entity of a single entity type. Since the segmentation causes empty or erroneous attribute values ​​in the segmented entity, it is necessary to perform abnormal correction on each segmented entity, wherein the correction is for each abnormal attribute in the segmented entity, and the correction may include operations such as error correction and missing completion.

[0054] All attributes in the corrected segmented entities have been corrected and missing-completion operations have been performed, and all corrected segmented entities are the results of cleaning the mixed entities.

[0055] Anomaly correction may refer to replacing the original erroneous value with the corresponding correct value or filling in the empty data. Among them, anomaly correction may adopt corresponding rules, models, trusted knowledge graphs, and trusted source data to match the original abnormal data with the correct value.

[0056] For example, for the first split entity and the second split entity split above, the name of the first split entity is Name1, the gender attribute is "male", and the medical attribute is empty; the name of the second split entity is also Name1, the gender attribute is empty, and the medical attribute is "gynecology". At this time, for the first split entity, the two attribute values ​​of Name1 and "male" are used to match the medical information as "male" from the trusted knowledge graph. Therefore, the "male" can be filled in the medical attribute of the first split entity as its attribute value. For the second split entity, the attribute value "gynecology" is used. Using the preset rules, it can be determined that the gender information is "female". Therefore, the "female" can be filled in the gender attribute of the second split entity as its attribute value.

[0057] The embodiment of the present application performs anomaly detection on the acquired data tuple to obtain an entity tuple characterized as abnormal and abnormal attributes in the entity tuple, and detects the entity type of the entity tuple. If it is detected that the entity type of the entity tuple is a mixed entity type, the entity tuple is determined to be a mixed entity. According to each abnormal attribute in the mixed entity and the corresponding attribute value, a split entity corresponding to each abnormal attribute is constructed, and the abnormality is corrected for each split entity to obtain a corrected split entity. All corrected split entities are determined to be the result of cleaning the mixed entity. By finding the mixed entity and constructing a split entity for each abnormal attribute in the mixed entity, the split entities are corrected and cleaned respectively, thereby avoiding data discard.

[0058] See also Figure 3 , is a flow chart of a method for cleaning data of mismatched entities provided in Example 3 of the present application. Figure 3 As shown, the detection of the entity type of the entity tuple in step S202 may further include the following steps:

[0059] Step S301, obtain a preset knowledge graph, and map and extract the objects of the entity tuple according to the preset knowledge graph to obtain an object set of the entity tuple.

[0060] In this embodiment, the preset knowledge graph may refer to graph data containing some known data and the association relationship between the data, wherein, based on the known data and the association relationship between the data, the entity object corresponding to the entity tuple may be obtained, thereby obtaining the object set corresponding to the entity tuple. In the mapping extraction, a part of the attribute information in the entity tuple is required, specifically the attribute information describing the characteristics of the entity tuple in the real world, such as the name of the country, etc.

[0061] Taking the country name as an example, the country name of entity A is Z. Using the country name Z, it can be mapped from the preset knowledge graph to the country corresponding to the country name. The country is currently a unique object. Therefore, there are no other objects for the entity A. For example, taking the name of a person as an example, the name of entity A is M. Using the name M, it can be mapped to 10 people from the preset knowledge graph, that is, the entity A corresponds to 10 objects. In actual use, in order to improve the accuracy, when mapping and extracting objects in the entity tuple, the name can be combined with at least one conflicting attribute to determine the corresponding object, such as the name M and the gender attribute, thereby further limiting the characteristics of the object and being able to match 3 objects, which is more accurate than the aforementioned 10 objects.

[0062] Step S302 , detecting whether the object set contains at least two objects. If it is detected that the object set contains at least two objects, determining that the entity type of the entity tuple is a mixed entity type.

[0063] In this embodiment, if the object set includes two or more objects, it means that the entity tuple has multiple entities in the real world. Therefore, the entity tuple is a mixed entity, that is, the corresponding entity type is a mixed entity type.

[0064] Of course, after detecting whether the object set contains at least two objects, if it is detected that the object set contains one object, the entity type of the entity tuple is determined to be a single entity type.

[0065] The embodiment of the present application provides a method for distinguishing abnormal attribute types based on a knowledge graph, which can effectively distinguish mixed entity categories and single entity categories, so as to implement different cleaning processes for entity tuples of different categories, so that the obtained cleaning results are more accurate.

[0066] See also Figure 4 , is a flow chart of a method for cleaning data of an incorrectly matched entity provided in Embodiment 4 of the present application. Figure 4 As shown, in the above step S201, abnormality detection is performed on the acquired data tuple to obtain an entity tuple characterized as abnormal and an abnormal attribute in the entity tuple, and the following steps may also be included:

[0067] Step S401, obtaining a data tuple and a preset anomaly detection rule.

[0068] In this embodiment, the preset anomaly detection rules are rules constructed based on the analysis of association relationships, relationships between conditions, etc., and may specifically include logic-based rules and model-based rules. The model may be a machine learning model, a neural network model, etc.

[0069] Logic-based rules can use rule discovery technology to analyze the relationship between data or conditions from existing data. They can also be rules manually specified according to needs. The rules can be embodied in the form of logical predicates. The form of the logical predicate is X->Y, which means that if condition X is met, then condition Y must be met; on the contrary, if condition X is met but Y is not met, this situation does not meet the rule, and the attribute corresponding to Y is an abnormal attribute. The use of logical predicates can also be combined with trusted source data Γ to better verify and match conditions X and Y.

[0070] The rule based on the machine learning model can be embodied in the form of a model predicate. The form of the model predicate is M(X,Y). Its function is to judge the correlation between attributes X and Y. If the correlation is lower than a certain threshold and attribute X is credible, then it is considered that the current entity tuple does not satisfy the rule, and the attribute corresponding to attribute Y is an abnormal attribute. The model predicate may not be combined with the source data Γ, but it may be necessary to rely on the source data Γ in the process of discovering the model predicate.

[0071] Step S402: for any entity tuple in the data tuple, use a preset anomaly detection rule to perform an anomaly detection on each attribute of any entity tuple, and obtain an abnormal attribute with an abnormal detection result.

[0072] Step S403: if any entity tuple has an abnormal attribute, then determine that any entity tuple is an entity tuple representing an abnormality, and determine the abnormal attribute belonging to the entity tuple from among all the abnormal attributes.

[0073] In this embodiment, anomaly detection includes the above-mentioned conflict anomaly, empty set anomaly, error anomaly, etc., and of course, other anomalies may also be included. If an entity tuple has abnormal attributes, it can be determined as an abnormal entity tuple, otherwise it is a normal entity tuple.

[0074] If an entity tuple is judged to be abnormal, all abnormal attributes in it need to be aggregated to form an abnormal attribute set, which can be classified according to different abnormalities. A conflicting abnormality corresponds to a conflicting attribute set, and so on. In subsequent use, the corresponding abnormal attribute set can be directly called according to different classifications, such as in step S203 of the above embodiment 2, the conflicting attribute set can be directly called.

[0075] In the embodiment of the present application, the abnormal attributes of the entity tuple can be detected based on the rules, which can cover a wider range of abnormal data and perform comprehensive detection, which helps to quickly and accurately realize abnormality detection, thereby facilitating subsequent cleaning operations.

[0076] See also Figure 5 , is a flow chart of a method for cleaning data of an incorrectly matched entity provided in Embodiment 5 of the present application, such as Figure 5 As shown, the above step S204 of performing abnormal correction on each segmented entity to obtain a corrected segmented entity may include the following steps:

[0077] Step S501 : for any segmented entity in each segmented entity, detect whether the attribute value of the segmented entity corresponding to the abnormal attribute is empty.

[0078] Step S502: if it is detected that the attribute value of the segmented entity corresponding to the abnormal attribute is not empty, the correct attribute value is matched according to the segmented entity and the corresponding abnormal attribute in combination with a preset matching rule and / or matching model.

[0079] Step S503: Use the correct attribute value to replace and correct the attribute value of the segmented entity corresponding to the abnormal attribute to obtain a corrected segmented entity.

[0080] In this embodiment, the exception correction is divided into replacement correction and completion correction, among which, the replacement correction means that there is an original attribute value under the corresponding exception attribute, and a matching attribute value needs to be used to replace the original attribute value; the completion correction means that the corresponding attribute value under the corresponding exception attribute is empty, and a matching attribute value needs to be used to fill the empty exception attribute.

[0081] Among them, the preset matching rule and the logic-based rule in step S401 of the above-mentioned embodiment 4 are rules of the same nature. Of course, the two can be the same rule. The preset matching model and the model in the model-based rule in step S401 of the above-mentioned embodiment 4 are models of the same nature. Of course, the two can be the same model.

[0082] It should be noted that the above logic-based rules and model-based rules can realize anomaly detection, and can also match the correct attribute value for each abnormal attribute during the anomaly detection process.

[0083] Optionally, after detecting whether the attribute value of the abnormal attribute corresponding to the segmentation entity is empty, the method further includes:

[0084] If it is detected that the attribute value of the abnormal attribute corresponding to the segmented entity is empty, the preset completion rules, completion knowledge graph and / or completion model are used to complete and correct the abnormal attribute corresponding to the segmented entity to obtain a corrected segmented entity.

[0085] Among them, the completion rule is a rule of the same nature as the above-mentioned logic-based rule, the completion model is a model of the same nature as the model in the above-mentioned model-based rule, and the completion knowledge graph is a knowledge graph of the same nature as the above-mentioned preset knowledge graph. Specifically, the rules, models, knowledge graphs defined in this embodiment can be the same as the rules, models, knowledge graphs, etc. in the aforementioned embodiments, and thus, the process of anomaly detection and subsequent matching of attribute values ​​can be implemented based on rules, models, and knowledge graphs. Of course, these rules, models, and knowledge graphs can be self-learned through continuous data accumulation in the process of executing the method of the present application to make themselves more accurate.

[0086] like Figure 6As shown, it is a framework schematic diagram of a cleaning method provided in Example 5 of the present application, wherein, for a data tuple containing N entity tuples, an abnormal entity tuple is obtained by anomaly detection, and then a mixed entity of a mixed entity type is obtained by type detection (a single entity of a single entity type can also be obtained), and then the corresponding entity tuple enters the process of splitting the entity (for mixed entities, single entities are not split), and the entity after splitting is corrected (single entities are directly corrected), and finally, the missing values ​​of the split and corrected entities are completed, and finally an entity tuple without abnormal attributes and missing attributes is obtained, that is, the cleaning of the entity tuple is realized. In this figure, the knowledge graph, rules, models, etc. can refer to the records in the specific embodiments, which act on the operation processes such as anomaly and type detection, splitting and correcting entities, and missing value completion. In addition, users can also be involved in each operation process, that is, user verification is performed to improve the accuracy of cleaning.

[0087] This embodiment can experiment with the above framework and obtain an effective cleaning method that integrates logic-based rules, knowledge graphs, and machine learning model-based rules. The accuracy of attribute error category discrimination (whether the entity tuple needs to be segmented or only needs error correction), the accuracy of attribute allocation and error correction, and the accuracy of missing value completion in this application are improved by 31.8%, 8.3%, and 39.5% respectively compared to existing error correction. In addition, compared to using only rule-based methods or only machine learning model-based methods, the accuracy can be improved by 35.5% and 30.3% respectively; at the same time, the execution efficiency of the method of this application has been improved to a certain extent, such as processing 1057217 entity tuples in only 1481 seconds.

[0088] See also Figure 7 , is a flow chart of a method for cleaning data of an incorrectly matched entity provided in Example 6 of the present application. Figure 7 As shown, after mapping and extracting the objects of the entity tuple in step S301 to obtain the object set of the entity tuple, the following steps may be included:

[0089] Step S701 , detecting whether the object set contains one object. If it is detected that the object set contains one object, determining that the entity type of the entity tuple is a single entity type.

[0090] Step S702 , determining that the entity tuple is a single entity, cleaning the single entity to obtain a cleaned single entity, wherein the entity tuple corresponding to the single entity type is represented as an object.

[0091] In this embodiment, as for the single entity type, it has been explained in the above embodiments and will not be repeated here.

[0092] See also Figure 8, is a flow chart of a method for cleaning data of an incorrectly matched entity provided in Embodiment 7 of the present application. Figure 8 As shown, the cleaning of the single entity in the above step S702 to obtain the cleaned single entity may include the following steps:

[0093] Step S801, setting the attribute value of each abnormal attribute in a single entity to empty.

[0094] Step S802: For any abnormal attribute whose attribute value is empty, a corresponding matching attribute value is matched according to the single entity and the abnormal attribute.

[0095] Step S803: fill the abnormal attributes with matching attribute values ​​to obtain a cleaned single entity.

[0096] In this embodiment, the exceptions in a single entity do not include conflict exceptions, which are generally empty set exceptions and error exceptions. For error exceptions, the attribute value of the exception attribute can be left empty and matched and filled together with the exception attribute of the empty set exception to obtain a cleaned single entity.

[0097] Of course, the empty set exception of a single entity can be matched and filled together with the empty set exception of the above-mentioned mixed entity, that is, after the above-mentioned mixed entity is split into split entities, the split entities are cleaned as a single entity.

[0098] Among them, in the process of matching the single entity and the abnormal attribute to the corresponding matching attribute value, the above rules, models and knowledge graphs can also be used for matching, corresponding to logic completion, knowledge graph completion and machine learning model completion respectively. The idea of ​​logic completion is to use logic-based rules to complete the missing attributes. For example, if the city of work is Shenzhen, then the province of work is Guangdong; knowledge graph completion is to use the known attributes in the entity tuple to match the entities in the knowledge graph G, and then use the information in G to complete the missing values ​​of the attributes in the entity tuple. For example, based on the partial information of a movie, the corresponding movie match can be found in the Internet Movie Database (IMDb), and then the missing movie information is completed with the information in IMDb. The input of the machine learning model completion is: a pre-trained model M, the known attribute value t[A] in the entity tuple t, the missing attribute B of t, and a set of possible values ​​of attribute B Cand(B). For each possible value val in Cand(B), the model M calculates the correlation strength score between t[A] and val, and finally takes the value val with the maximum correlation score to complete t[B].

[0099] It should be known that among the above three completion strategies, the first two (i.e., logic completion and knowledge graph completion) are preferred because they have higher credibility. When the first two strategies fail, machine learning model completion can be used to fill in missing values.

[0100] Corresponding to the method for cleaning data of mismatched entities in the above embodiment, Fig. 9 The structural block diagram of the cleaning device for the data of the mismatched entity provided in the eighth embodiment of the present application is shown. The cleaning device is applied to Figure 1 In the server, the user corresponding to the client sends the data to be cleaned (ie, data tuple) to the server to trigger the server to perform the cleaning task. For ease of description, only the part related to the embodiment of the present application is shown.

[0101] See also Fig. 9 , the cleaning device comprises:

[0102] Anomaly detection module 91, used to perform anomaly detection on the acquired data tuples to obtain entity tuples characterized as abnormal and abnormal attributes in the entity tuples, wherein the data tuples include at least one entity tuple;

[0103] A type detection module 92 is used to detect the entity type of the entity tuple, and if it is detected that the entity type of the entity tuple is a mixed entity type, the entity tuple is determined to be a mixed entity, wherein the entity tuple corresponding to the mixed entity type represents more than one object;

[0104] An entity segmentation module 93, for constructing a segmentation entity corresponding to each abnormal attribute according to each abnormal attribute and the corresponding attribute value in the mixed entity, wherein each segmentation entity includes an abnormal attribute and a corresponding attribute value;

[0105] The mixed entity cleaning module 94 is used to perform abnormal correction on each segmented entity to obtain a corrected segmented entity, and determine that all corrected segmented entities are the results of cleaning the mixed entity.

[0106] Optionally, the type detection module 92 includes:

[0107] A knowledge graph mapping unit is used to obtain a preset knowledge graph, and map and extract the objects of the entity tuple according to the preset knowledge graph to obtain an object set of the entity tuple;

[0108] The type detection unit is used to detect whether the object set contains at least two objects. If it is detected that the object set contains at least two objects, it is determined that the entity type of the entity tuple is a mixed entity type.

[0109] Optionally, the anomaly detection module 91 includes:

[0110] An acquisition unit, used for acquiring data tuples and preset anomaly detection rules;

[0111] An anomaly detection unit is used to perform an anomaly detection on each attribute of any entity tuple in the data tuple using a preset anomaly detection rule to obtain an anomaly attribute with an abnormal detection result;

[0112] The abnormality determination unit is used to determine any entity tuple as an entity tuple representing an abnormality if an abnormal attribute exists in any entity tuple, and to determine the abnormal attribute belonging to the entity tuple from all the abnormal attributes.

[0113] Optionally, the entity cleaning module 94 includes:

[0114] An attribute value detection unit, used for detecting, for any segmented entity in each segmented entity, whether an attribute value of an abnormal attribute corresponding to the segmented entity is empty;

[0115] A first attribute value matching unit is used to match the correct attribute value according to the segmented entity and the corresponding abnormal attribute in combination with a preset matching rule and / or matching model if it is detected that the attribute value of the segmented entity corresponding to the abnormal attribute is not empty;

[0116] The attribute value replacement unit is used to replace and correct the attribute value of the segmented entity corresponding to the abnormal attribute with the correct attribute value to obtain the corrected segmented entity.

[0117] Optionally, the entity cleaning module 94 further includes:

[0118] The attribute value completion unit is used to, after detecting whether the attribute value of the abnormal attribute corresponding to the segmented entity is empty, use the preset completion rules, completion knowledge graph and / or completion model to complete and correct the abnormal attribute corresponding to the segmented entity to obtain the corrected segmented entity if it is detected that the attribute value of the abnormal attribute corresponding to the segmented entity is empty.

[0119] Optionally, the cleaning device further comprises:

[0120] A single entity type determination module is used to determine that the entity type of the entity tuple is a single entity type if it is detected that the object set contains one object after mapping and extracting the object of the entity tuple to obtain the object set of the entity tuple;

[0121] The single entity cleaning module is used to determine that the entity tuple is a single entity, clean the single entity, and obtain the cleaned single entity, wherein the entity tuple corresponding to the single entity type is represented as an object.

[0122] Optionally, a single entity cleaning module includes:

[0123] An attribute value preprocessing unit, used for setting the attribute value of each abnormal attribute in a single entity to empty;

[0124] A second attribute value matching unit is used to match a corresponding matching attribute value according to a single entity and the abnormal attribute for any abnormal attribute whose attribute value is empty;

[0125] The attribute value filling unit is used to fill the abnormal attributes with matching attribute values ​​to obtain a cleaned single entity.

[0126] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0127] Fig.10 This is a schematic diagram of the structure of a computer device provided in Example 9 of the present application. Fig.10 As shown, the computer device of this embodiment includes: at least one processor ( Fig.10 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in the above-mentioned method for cleaning data of any of the mismatched entities are implemented.

[0128] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Fig.10 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0129] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0130] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments may also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory may also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of computer programs, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0131] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0132] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a computer device, the steps in the above-mentioned method embodiment can be implemented when the computer device executes it.

[0133] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0134] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0135] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for cleaning data of mismatched entities, characterized in that: The cleaning method comprises: Obtaining a data tuple and a preset anomaly detection rule, wherein the data tuple includes at least one entity tuple, the entity tuple is a row of data in a relational table, the row of data includes at least one column, and each column represents an attribute value of an attribute of a corresponding entity; For any entity tuple in the data tuple, using the preset anomaly detection rule, perform anomaly detection on each attribute of the any entity tuple to obtain an abnormal attribute whose detection result is abnormal; If the abnormal attribute exists in any one of the entity tuples, determining the any one of the entity tuples as an entity tuple representing an abnormality, and determining the abnormal attribute belonging to the entity tuple from all the abnormal attributes; Obtain a preset knowledge graph, and map and extract the objects of the entity tuple according to the preset knowledge graph to obtain an object set of the entity tuple; Detecting whether the object set contains at least two objects, if it is detected that the object set contains at least two objects, determining that the entity type of the entity tuple is a mixed entity type, if it is detected that the entity type of the entity tuple is a mixed entity type, determining that the entity tuple is a mixed entity, wherein the entity tuple corresponding to the mixed entity type represents more than one object, and the object is an object or a place; If the abnormal attribute of the mixed entity is a conflict abnormality, the mixed entity is segmented into segmented entities corresponding to different objects or locations, wherein each segmented entity includes an abnormal attribute and a corresponding attribute value; Performing abnormality correction on each segmented entity to obtain a corrected segmented entity, and determining that all corrected segmented entities are the results of cleaning the mixed entity; The abnormality correction is performed on each segmented entity to obtain a corrected segmented entity, including: For any segmented entity in each segmented entity, detecting whether the attribute value of the abnormal attribute corresponding to the segmented entity is empty; If it is detected that the attribute value of the abnormal attribute corresponding to the segmented entity is not empty, matching the correct attribute value according to the segmented entity and the corresponding abnormal attribute in combination with a preset matching rule and / or matching model; The correct attribute value is used to replace and correct the attribute value of the segmented entity corresponding to the abnormal attribute to obtain a corrected segmented entity.

2. The cleaning method according to claim 1, characterized in that: After detecting whether the attribute value of the segmented entity corresponding to the abnormal attribute is empty, the method further includes: If it is detected that the attribute value of the abnormal attribute corresponding to the segmented entity is empty, the preset completion rule, completion knowledge graph and / or completion model are used to complete and correct the abnormal attribute corresponding to the segmented entity to obtain the corrected segmented entity.

3. The cleaning method according to claim 1, characterized in that: After mapping and extracting the objects of the entity tuple to obtain the object set of the entity tuple, the method further includes: Detecting whether the object set contains one object, and if it is detected that the object set contains one object, determining that the entity type of the entity tuple is a single entity type; It is determined that the entity tuple is a single entity, and the single entity is cleaned to obtain a cleaned single entity, wherein the entity tuple corresponding to the single entity type is represented as an object.

4. The cleaning method according to claim 3, characterized in that: The step of cleaning the single entity to obtain a cleaned single entity comprises: Setting the attribute value of each abnormal attribute in the single entity to empty; For any abnormal attribute whose attribute value is empty, a corresponding matching attribute value is matched according to the single entity and the abnormal attribute; The abnormal attributes are filled using the matching attribute values ​​to obtain a cleaned single entity.

5. A device for cleaning data of mismatched entities, characterized in that: The cleaning device comprises: Anomaly detection module, including: An acquisition unit, configured to acquire a data tuple and a preset anomaly detection rule, wherein the data tuple includes at least one entity tuple, the entity tuple is a row of data in a relational table, the row of data includes at least one column, and each column represents an attribute value of an attribute of a corresponding entity; an anomaly detection unit, configured to perform an anomaly detection on each attribute of any entity tuple in the data tuple using the preset anomaly detection rule, and obtain an anomaly attribute whose detection result is abnormal; an abnormality determination unit, configured to determine, if the abnormal attribute exists in any one of the entity tuples, that the any one of the entity tuples is an entity tuple representing an abnormality, and to determine, from all the abnormal attributes, an abnormal attribute belonging to the entity tuple; Type detection module, including: A knowledge graph mapping unit, used to obtain a preset knowledge graph, and map and extract the objects of the entity tuple according to the preset knowledge graph to obtain an object set of the entity tuple; a type detection unit, configured to detect whether the object set includes at least two objects, and if it is detected that the object set includes at least two objects, determine that the entity type of the entity tuple is a mixed entity type, and if it is detected that the entity type of the entity tuple is a mixed entity type, determine that the entity tuple is a mixed entity, wherein the entity tuple corresponding to the mixed entity type represents more than one object, and the object is an object or a place; An entity segmentation module, configured to segment the mixed entity into segmented entities corresponding to different objects or locations if the abnormal attribute of the mixed entity is a conflict abnormality, wherein each segmented entity includes an abnormal attribute and a corresponding attribute value; A mixed entity cleaning module, used to perform abnormal correction on each segmented entity to obtain a corrected segmented entity, and determine that all corrected segmented entities are the results of cleaning the mixed entity; The entity cleaning module comprises: An attribute value detection unit, used for detecting, for any segmented entity in each segmented entity, whether an attribute value of an abnormal attribute corresponding to the segmented entity is empty; A first attribute value matching unit, configured to match a correct attribute value according to the segmented entity and the corresponding abnormal attribute in combination with a preset matching rule and / or matching model if it is detected that the attribute value of the abnormal attribute corresponding to the segmented entity is not empty; The attribute value replacement unit is used to use the correct attribute value to replace and correct the attribute value of the segmented entity corresponding to the abnormal attribute to obtain a corrected segmented entity.

6. A computer device, characterized in that: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the cleaning method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the cleaning method according to any one of claims 1 to 4 is implemented.

Images