Data processing method, device and storage medium

Abstract

Embodiments of the present application provide a data processing method and device and a storage medium. In the embodiments of the present application, the similarity between data objects corresponding to a plurality of data records is determined according to the influence degree of the data object attributes on the same data object recognition result and the attribute information of the data objects; and the data records belonging to the same data object are determined according to the similarity between the data objects corresponding to the plurality of data records, thereby realizing the recognition of the data records of the same object. Since the influence degree of the data object attributes on the same data object recognition result is considered when determining the similarity between the data objects corresponding to the plurality of data records, the accuracy of subsequent same data object recognition is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus and storage medium. Background Technology

[0002] With the development of the information age, the media carrying various information are becoming increasingly diverse, making effective data management increasingly important. In existing technologies, various reasons lead to multiple identifiers for the same target object, which can easily cause incomplete data when building a knowledge base for that target object. To build a knowledge base for a target object, it is necessary to identify data belonging to the same object from massive amounts of data. Summary of the Invention

[0003] This application provides a data processing method, apparatus, and storage medium to achieve data record identification of the same object and improve the accuracy of the identification of the same object.

[0004] This application provides a data processing method, including: Retrieve multiple data records; the multiple data records include attribute information of data objects; Based on the attribute information of the data objects contained in the multiple data records, and the influence degree of each attribute of the data objects, the similarity between the data objects corresponding to the multiple data records is determined. Based on the similarity between the data objects corresponding to the multiple data records, determine the data records belonging to the same data object from the multiple data records; Based on the data records of the same data object, construct the association network of the same data object.

[0005] This application embodiment also provides a computing device, including: a memory and a processor; wherein, the memory is used to store computer programs; The processor is coupled to the memory and is used to execute the computer program to perform the steps in the data processing method described above.

[0006] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the above-described data processing method.

[0007] In this embodiment, the similarity between data objects corresponding to multiple data records can be determined based on the influence of data object attributes on the identification results of the same data object, and the attribute information of the data object; and based on the similarity between the data objects corresponding to multiple data records, data records belonging to the same data object can be identified, thus realizing the identification of data records belonging to the same object. Since the influence of data object attributes on the identification results of the same data object is considered when determining the similarity between data objects corresponding to multiple data records, it helps to improve the accuracy of subsequent identification of the same data object. Attached Figure Description

[0008] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 and Figure 2 A flowchart illustrating the data processing method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the construction process of the association network of data objects provided in the embodiments of this application; Figure 4 A schematic diagram illustrating the update process of the associated network of data objects provided in the embodiments of this application; Figure 5 and Figure 6 A schematic diagram of the main index node of the connected subgraph of the data object provided in the embodiments of this application; Figure 7 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] In this embodiment, to construct a network of associations for data objects, the similarity between data objects corresponding to multiple data records can be determined based on the influence of data object attributes on the identification results of data objects, and the attribute information of each data object. Furthermore, based on the similarity between the data objects corresponding to multiple data records, data records belonging to the same data object are identified, thus achieving the identification of data records belonging to the same object. Since the influence of data object attributes on the identification results of data objects is considered when determining the similarity between data objects corresponding to multiple data records, it helps improve the accuracy of subsequent identification of data objects. On the other hand, when identifying data records belonging to the same data object, the comprehensive consideration of various attributes of the data object helps improve the completeness and comprehensiveness of the data record collection for the same data object, thereby helping to improve the comprehensiveness and completeness of the subsequently constructed network of associations for data objects.

[0011] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0012] It should be noted that the same reference numerals denote the same object in the following figures and embodiments. Therefore, once an object is defined in one figure or embodiment, it does not need to be discussed further in subsequent figures and embodiments.

[0013] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application. Figure 1 As shown, the method mainly includes: 101. Retrieve multiple data records; multiple data records include the attribute information of the data objects.

[0014] 102. Based on the attribute information of the data objects contained in multiple data records, and the influence degree of each attribute of the data objects, determine the similarity between the data objects corresponding to the multiple data records.

[0015] 103. Based on the similarity between the data objects corresponding to multiple data records, determine the data records that belong to the same data object from the multiple data records.

[0016] 104. Based on the data records of the same data object, construct a network of associations for the same data object.

[0017] In this embodiment, the data object is an objectively existing entity in nature, which is the object of study in this application embodiment. In this application embodiment, in order to have as comprehensive an understanding of the data object as possible, a knowledge base of the data object can be constructed based on the data records of the data object. The data object varies depending on the application scenario and research purpose.

[0018] For example, in product maintenance scenarios, a product maintenance knowledge base can be built based on the product's maintenance records, providing reference materials for maintenance personnel. Correspondingly, the data object can be the relevant product. A product refers to anything used and consumed by people that satisfies some need; it can be physical equipment such as mechanical equipment, electronic equipment, optical equipment, or opto-mechatronic devices, or computer program products such as applications and websites. Correspondingly, the data records can be product maintenance records, etc.

[0019] For example, in a healthcare setting, a patient health record knowledge base can be built based on a patient's medical records, providing reference materials for medical staff to understand the patient's condition. Accordingly, the data object can be the patient, and the data records can be the patient's medical registration records, etc.

[0020] For example, in online shopping scenarios, a user experience knowledge base for products can be built based on product review data, providing other consumers with more detailed product descriptions. Accordingly, the data object can be the product itself, and the data record can be product review data, etc. In this embodiment, the product is the result of labor produced for sale, including physical goods and potentially virtual services, etc.

[0021] Regardless of the application scenario, to build a knowledge base for data objects, it is necessary to identify data records belonging to the same data object from massive amounts of data, and then construct the knowledge base for that data object based on these records. Therefore, in step 101, multiple data records can be obtained. "Multiple records" refers to two or more records; in this embodiment, "multiple records" primarily refers to a massive amount of data records. These multiple data records include attribute information of the data objects. Each data record includes attribute information of the data object. In this embodiment, the data objects contained in the multiple data records are corresponding research objects, which may be the same data object or different data objects.

[0022] In this embodiment, the attribute information of a data object refers to the information corresponding to the attributes of the data object. It is information that reflects the nature of the data object to a certain extent, and this attribute information can distinguish different data objects to some degree. For example, the attribute information of a data object may be its basic information, performance information, or other information. In this embodiment, the attribute information of a data object can be understood as the attribute values ​​corresponding to the attributes of the data object, which are the specific attribute content.

[0023] The attribute information of data objects differs across application scenarios. For example, in the product operation and maintenance scenario mentioned above, the attribute information of data objects can be product attribute information, including but not limited to: basic product information, performance parameter information, etc. For instance, product attribute information may include one or more of the following: product identifier (ID), product name, product model, manufacturing date, manufacturing address, manufacturer, and applicable standards. "Multiple" refers to two or more types. The product identifier can be a product identification code, such as a barcode number.

[0024] For example, in the aforementioned medical scenario, the data object could be the patient. The patient's attribute information could be their basic information, such as personal information.

[0025] Because the attributes of data objects can distinguish different data objects to a certain extent, and different attributes have varying degrees of distinguishing power, their influence on the identification of the same data object varies. Specifically, the attribute with the greater degree of distinguishability between different data objects has a greater influence on the identification of the same data object. For example, for products, a single product typically has a unique product identifier, while different products may share the same product name. The same manufacturer may also produce different products. Therefore, the product identifier, product name, and manufacturer have different degrees of distinguishability between different products, and consequently, their influence on the identification results of the same product varies. Generally, the product identifier has a greater influence on the identification results of the same product than the product name, and the product name has a greater influence on the identification results of the product than the manufacturer.

[0026] Based on this, in step 102, the similarity between data objects corresponding to multiple data records can be determined according to the attribute information of the data objects contained in multiple data records and the influence degree corresponding to each attribute of the data objects. The influence degree corresponding to an attribute of a data object refers to the influence of that attribute on the identification result of the same data object. The influence degree corresponding to the same attribute is the same and does not differ due to differences in the attribute information (i.e., attribute value) corresponding to that attribute.

[0027] In this application embodiment, the specific implementation method for determining the influence degree corresponding to the attributes of a data object is not limited. In some embodiments, the influence degree of each attribute of the data object can be obtained through a large number of preliminary verification experiments. Alternatively, machine learning methods can be used to train the influence degree of the attributes of the data object on the identification results of the same data object, thereby obtaining the influence degree corresponding to each attribute of the data object. For example, multiple data records known to be the same data object can be obtained; an initial influence degree can be set for each attribute of the data object; further, the influence degree corresponding to each attribute of the data object can be adjusted with the goal of minimizing the error between the similarity and the true similarity value between the data objects corresponding to the multiple data records of the same data object, until the error between the similarity and the true similarity value between the data objects corresponding to the multiple data records of the same data object meets the set requirements. The influence degree when the error between the similarity and the true similarity value between the data objects corresponding to the multiple data records of the same data object meets the set requirements is taken as the influence degree corresponding to each attribute of the data object in step 102. In this application embodiment, the true similarity value between the data objects corresponding to the data records of the same data object can be 1.

[0028] In some embodiments, step 102 can be implemented as follows: For any two data records A and B among multiple data records, determine the similarity between the data objects corresponding to these two data records A and B based on the attributes of the data objects contained in the two data records A and B, and the influence degree corresponding to each attribute of the data objects. That is, calculate the similarity between the data objects corresponding to each pair of data records among multiple data records.

[0029] Because the amount of data in multiple data records is large, sometimes even massive, the computational workload for determining the similarity between pairs of data records corresponding to data objects is substantial, resulting in low data processing efficiency. In some embodiments, to improve data processing efficiency, the multiple data records in step 101 can be clustered according to certain rules, grouping data records with a higher probability of belonging to the same data object into a single cluster (i.e., ...). Figure 2 The data records in the data are clustered, and then similarity is calculated for the data records in each cluster.

[0030] Since the attributes of data objects have varying degrees of influence, if data objects in different data records have the same attribute that significantly impacts the identification result of the same data object, then these data records are more likely to belong to the same data object. Based on this, key attribute information corresponding to each data record can be obtained from the attribute information of the data objects contained in multiple data records. The key attributes are the top N attributes whose influence on the identification result of the same data object is ranked. N is a positive integer and less than the number of attributes of the data object. For example, N is 2 or 3, etc.

[0031] Different data objects have different key attributes. In the product operation and maintenance scenario mentioned above, the key attributes of the product may be the product identifier and / or product name. In the medical scenario mentioned above, the key attributes of the patient may be the patient's identity identifier or a specific name symbol, etc.

[0032] After obtaining the key attribute information of multiple data records, the data records can be clustered based on the key attribute information corresponding to each record to obtain at least one cluster (corresponding to...). Figure 2 (Data record clustering in the data).

[0033] In this application, the specific implementation method for clustering multiple data records is not limited. Since data records may contain errors or be invalid during generation or entry, to improve clustering accuracy and flexibility, when clustering multiple data records, the key attribute information of any data record A can be validated to determine the valid key attribute information corresponding to data record A. The valid key attribute information for all data records can be determined in the same manner.

[0034] In practical applications, key attribute information often has a defined number of valid bits and data organization rules. The number of valid bits in key attribute information refers to the number of valid characters within the key attribute information. Data organization rules refer to the way data is organized according to certain methods and rules. The number of valid bits in key attribute information must meet the requirements for the number of valid bits in a key attribute, and the data organization rules for the key attribute information must also meet the requirements for the data organization rules of that key attribute. Therefore, based on the number of valid bits and the data organization rules of the key attribute, the validity of the key attribute information corresponding to any data record A can be checked to determine the valid key attribute information corresponding to any data record A.

[0035] Specifically, based on the data organization rules of key attributes, characters that do not conform to the data organization rules can be deleted from the key attribute information of data record A to obtain the valid bits of the key attribute information corresponding to data record A. For example, if the key attribute information consists of numbers and letters, then non-number and non-letter characters can be deleted from the key attribute information of data record A. As another example, if the key attribute information consists of letters and / or Chinese characters, then non-number and non-Chinese characters can be deleted from the key attribute information of data record A.

[0036] Furthermore, it can be determined whether the number of valid bits in the key attribute information corresponding to data record A is greater than or equal to the number of valid bits in the corresponding key attribute. If the determination result is yes, then the key attribute information corresponding to data record A is determined to be valid key attribute information.

[0037] For example, if the key attribute is "product name," we can determine whether the number of valid characters in the product name in data record A is greater than or equal to the number of valid characters in the product name. If the number of valid characters in the product name in data record A is greater than or equal to the number of valid characters in the product name, then the product name in data record A is determined to be a valid product name. For example, if the product name is "computer," it can also be called "computer," etc. The number of valid characters in the product name can be 2. Therefore, if the number of characters in the product name in data record A is greater than or equal to 2, then the product name in data record A is determined to be a valid product name.

[0038] For example, the key attribute is the product identifier. If the product identifier is known to consist of letters and / or numbers of a set length, such as 15 or 18 digits, then it can be determined whether the number of valid digits in the product identifier in data record A is greater than or equal to the set length. If the result is yes, the product identifier in data record A is determined to be a valid product identifier.

[0039] In practical applications, key attribute information also needs to meet certain data organization rules. For example, some key attribute information may include data bits and check bits. Therefore, if the number of valid bits in the key attribute information of data record A is greater than or equal to the number of valid bits in the corresponding key attribute, the validity of the valid bits of the key attribute information corresponding to data record A can be validated according to the data organization rules of the key attributes. For example, the data organization rules of key attributes include: the valid bits of the set priority position need to be in the set data format. It can be determined whether the valid bits of the set priority position in data record A are in the set data format. If the determination result is yes, the key attribute information corresponding to data record A is determined to be valid key attribute information.

[0040] For example, the data organization rule for a key attribute is as follows: the key attribute information includes data bits and check bits, and the check bits are obtained according to a set check bit generation algorithm. Then, the data bits and check bits of the key attribute in data record A can be obtained from the key attribute information corresponding to data record A; and a check bit can be generated according to the set check bit generation algorithm; then, it can be determined whether the generated check bit is the same as the check bit of the key attribute in data record A; if they are the same, the key attribute information corresponding to data record A is determined to be valid key attribute information. Conversely, if the generated check bit is different from the check bit of the key attribute in data record A, the key attribute information corresponding to data record A is determined to be invalid key attribute information.

[0041] The above-described implementation methods for validating key attribute information are merely illustrative and do not constitute a limitation. In the embodiments of this application, the validity detection methods shown in the above embodiments can be implemented selectively or in combination, depending on the nature of the key attribute. For example, in a product operation and maintenance scenario, if the key attribute is the product name, validity can be verified based on the number of valid digits of the product name in the operation and maintenance record; if the key attribute is the product identifier, validity can be verified based on the number of valid digits of the product identifier in the operation and maintenance record and the data organization rules of the product identifier.

[0042] For example, in a medical setting, if the key attribute is the patient's identification, validity can be verified based on the number of valid digits of the patient's identification in the medical record and the data organization rules for the patient's identification. Similarly, for the key attribute of the patient's name symbol, validity can be verified based on the number of valid digits of the patient's name symbol in the medical record.

[0043] Considering that attribute information may be subject to errors during data record generation or entry, for various reasons—such as changes in product identification rules within a domain leading to previous identification rules no longer conforming to current rules, or pronunciation issues causing data entry errors or the input of similar-sounding words—after determining the key validity attributes for each data record, error tolerance processing can be applied to the valid key attribute information for multiple data records based on the corresponding error tolerance methods. This results in error-tolerant valid key attribute information for multiple data records.

[0044] In some embodiments, data supplementation can be performed on the valid key attribute information corresponding to multiple data records according to the data organization rules of key attributes, so as to obtain valid key information that conforms to the data organization rules, which serves as the valid key attribute information after fault tolerance processing for multiple data records. For example, in a medical scenario, if the data organization rules for patient identification change, such as identifications from a certain era lacking a check digit, a check digit can be generated using the identification in the data record according to the check digit generation method corresponding to the data organization rules of identification; and the check digit can be supplemented to the position required by the data organization rules of identification to obtain a valid identification that conforms to the data organization rules.

[0045] In some other embodiments, considering that there may be accent errors of the narrator, or the recorder may record the information as other characters with the same pronunciation, phonetic symbols that are prone to errors can be counted in advance. For example, in Chinese pronunciation, there are flat tongue sounds, retroflex sounds, front and back nasals, etc. Based on the phonetic symbols set in the pre-statistics, the effective key attribute information corresponding to multiple data records can be converted into phonetic notation; and the phonetic symbols set in the phonetic notation of the effective key attribute information of multiple data records are removed, so as to obtain the effective phonetic notation of the effective key attribute information of multiple data records, which is used as the effective key attribute information after fault-tolerant processing corresponding to multiple data records. For example, for the Chinese characters "Zhang San", it can be converted into "Zhangsan", and the retroflex "h" and the back nasal "g" in "Zhangsan" are removed to obtain "Zansan", etc.

[0046] The manner of performing fault-tolerant processing on key attributes shown in the above embodiments is only an exemplary description, and does not constitute a limitation. In actual use, different fault-tolerant processing methods can be implemented either alternatively or in combination.

[0047] After performing fault-tolerant processing on the effective key attribute information corresponding to multiple data records, the multiple data records can be clustered according to the effective key attribute information after fault-tolerant processing corresponding to the multiple data records. Optionally, according to the effective key attribute information after fault-tolerant processing corresponding to multiple data records, data records having the same effective key attribute information after fault-tolerant processing among the multiple data records can be clustered into one cluster.

[0048] Specifically, multiple data records can be clustered sequentially according to the order of influence degree corresponding to the key attributes of the data objects. For two key attributes X and Y in two adjacent influence order positions, based on the effective key attribute information after fault-tolerant processing corresponding to key attribute X with higher influence degree, data records with the same effective key attribute information after fault-tolerant processing corresponding to key attribute X in the multiple data records can be divided into the same cluster; for data records with different effective key attribute information after fault-tolerant processing corresponding to key attribute X, based on the effective key attribute information after fault-tolerant processing corresponding to key attribute Y with lower influence degree, data records with the same effective key attribute information after fault-tolerant processing corresponding to key attribute Y in the multiple data records are divided into the same cluster.

[0049] For example, if the key attributes are data object identifier and data object name, and the influence of the data object identifier is greater than that of the data object name, then data records with the same data object identifier after fault tolerance processing can be first grouped into the same cluster; then data records with the same data object name after fault tolerance processing can be grouped into the same cluster, thereby achieving clustering of multiple data records based on the key attribute information of multiple data records to obtain at least one cluster.

[0050] For any cluster D, the similarity between corresponding data objects in cluster D can be calculated based on the attribute information of the data objects contained in the data records of any cluster D, and the influence degree of each attribute of the data objects. Figure 2 (Pairwise similarity calculation within the same cluster). In this way, when calculating the similarity between data objects corresponding to data records, only the data records within the same cluster need to be similarized, without having to calculate the similarity of all data records in step 101 pairwise. This reduces the amount of data for similarity calculation and improves the efficiency of similarity calculation.

[0051] In this application embodiment, the specific implementation method for calculating the similarity between data objects corresponding to data records in a cluster is not limited. In some embodiments, for any two data records A and B in any cluster D, the similarity of the data objects corresponding to the attribute P1 can be determined based on the influence degree corresponding to any attribute P1 and the attribute information corresponding to the attribute P1 in the two data records A and B.

[0052] Optionally, validity checks can be performed on the attribute information corresponding to any attribute P1 in any two data records A and B to determine the validity of attribute P1 in any two records. For a description of validity checks on key attribute information, please refer to the relevant content in the above embodiments, which will not be repeated here. The following is an exemplary description of an implementation method for validity checks on attribute information of attributes other than key attributes.

[0053] In some embodiments, it can be determined whether the number of valid bits of the attribute information corresponding to attribute P1 contained in the data record is greater than or equal to the valid bit threshold corresponding to attribute P1; if the determination result is yes, then attribute P1 in the data record is determined to be valid; otherwise, attribute P1 in the data record is determined to be invalid.

[0054] For example, attribute P1 represents the creation date of a data object, such as the production date or launch date of a product. If the creation date in a data record has fewer than 4 valid digits, the creation date of the data object in the data record is considered invalid. If the creation date in a data record has more than 4 valid digits, the creation date can be identified by year, month, and day. If the year and / or month are missing from the creation date of the data object in the data record, the creation date of the data object in the data record is considered invalid. If the creation date in a data record has more than 4 valid digits, and both the year and month are present, the creation date of the data object in the data record is considered valid.

[0055] For example, attribute P1 represents the production address of a data object. If the production address of the data object contained in the data record has fewer than 4 valid bits, the production address of the data object in the data record is determined to be invalid. If the production date contained in the data record has more than 4 valid bits, the production address of the data object in the data record is determined to be valid.

[0056] For example, attribute P1 represents the manufacturer name of a data object. If the manufacturer name of the data object contained in a data record has fewer than two valid digits, the manufacturer name of the data object in the data record is determined to be invalid. If the generation date contained in a data record has more than two valid digits, the manufacturer name of the data object in the data record is determined to be valid.

[0057] The methods for validating the attribute information of data records shown in the above embodiments are merely illustrative and do not constitute a limitation.

[0058] After determining the validity of any attribute P1 in any two data records A and B in the same cluster, the similarity of the data objects corresponding to data records A and B in attribute P1 can be determined based on the validity of attribute P1 in data records A and B, the attribute information corresponding to attribute P1, and the influence degree corresponding to attribute P1.

[0059] In some embodiments, attribute P1 is a data object identifier. Since a data object identifier can uniquely identify a data object, it can be determined that the influence of the data object identifier on the identification result of the same data object is 100%. Therefore, for data records A and B with the same data object identifier, the similarity between the data objects corresponding to data records A and B can be determined to be a set value, such as 100.

[0060] Accordingly, if both data object identifiers in data records A and B are valid, it can be determined whether the data object identifiers in the two data records A and B are the same; if the determination result is yes, the influence degree corresponding to the data object identifier is used as the similarity of the data objects corresponding to the two data records in terms of data object identifiers. Conversely, if any two data records A and B contain invalid data object identifiers, or if any two data records A and B contain different data object identifiers, the similarity of the data objects corresponding to the two data records in any attribute can be determined based on the validity of other attributes besides the data object identifier and the influence degree corresponding to those attributes.

[0061] Specifically, for any attribute P1 other than the data object identifier, if there are invalid instances of attribute P1 in data records A and B, the weight of the influence corresponding to attribute P1 can be determined based on the number of valid instances of attribute P1 in data records A and B. The number of valid instances of attribute P1 in data records A and B refers to the number of valid attribute information corresponding to attribute P1 in data records A and B. For example, if attribute P1 is a product name; the product name in data record A is valid, but the product name in data record B is invalid, then the number of valid instances of the product name is 1.

[0062] Then, based on the influence of attribute P1 on the identification results of the same data object and the weight of the influence corresponding to attribute P1, the similarity of the data objects corresponding to data records A and B in terms of attribute P1 can be determined. For example, if the number of valid attributes P1 in data records A and B is 0, that is, attribute P1 is invalid in both data records A and B, the weight of the influence corresponding to attribute P1 is determined to be the set first weight value. If the number of valid attributes P1 in both data records A and B is 1, that is, one attribute P1 is valid and the other is invalid, the weight of the influence corresponding to attribute P1 can be determined to be the set second weight value. Where 0 ≤ first weight value < second weight value < 1. For example, the first weight value is 0.1, the second weight value is 0.2, etc.

[0063] Accordingly, if attribute P1 is valid in both data records A and B, the weight of the influence corresponding to attribute P1 can be determined based on the attribute information corresponding to attribute P1 in data records A and B; then, the similarity of the data objects corresponding to attribute P1 in data records A and B can be determined based on the influence corresponding to attribute P1 and the weight of the influence corresponding to attribute P1.

[0064] In some optional examples, attribute P1 is a time attribute, such as the production date of a data object. The weight of the influence of time on the identification result of the same data object can be determined based on the time information and its granularity in data records A and B. The granularity of the time information refers to the precise time unit. For example, if the time information is a specific year, month, and day, then the granularity is accurate to the day. Or, if the time information is a specific year, month, day, and hour, then the granularity is accurate to the hour, and so on.

[0065] Optionally, if the granularity of the time information of data objects in data records A and B is the same, then it is determined whether the time information in data records A and B is the same; if the time information of data objects in data records A and B is the same, the weight of the influence corresponding to time is determined to be 1. Correspondingly, if the time information of data objects in data records A and B is different, the weight of the influence of time on the identification result of the same data object is determined to be 0, that is, the weight of the influence corresponding to the event is 0.

[0066] If the granularity of the time information for data objects in data records A and B is different, the first time information with the lower granularity in data records A and B can be compared to see if the time corresponding to the granularity of the first time information is the same as that of the other time information. If they are the same, the weight of the influence of time is determined to be a set third weight value. 0 < third weight value < 1. For example, the third weight value is 0.5. Correspondingly, if the time corresponding to the granularity of the first time information is different from that of the other time information, the weight of the influence of time on the identification result of the same data object is determined to be 0.

[0067] In some alternative examples, attribute P1 is an attribute other than the data object identifier and the time attribute. The character similarity of the attribute information corresponding to attribute P1 in two data records A and B can be determined based on the attribute information corresponding to attribute P1 in the two data records A and B. And the weight of the influence corresponding to attribute P1 can be determined based on the character similarity of the attribute information corresponding to attribute P1 in the two data records A and B.

[0068] Optionally, if the attribute information of any attribute P1 is purely numeric, the purely numeric attribute information of attribute P1 in two data records A and B can be compared sequentially to obtain the number of different numbers for attribute P1. The character similarity of the attribute information corresponding to attribute P1 in two data records A and B is measured by the number of different numbers for attribute P1 in two data records A and B. The more different numbers for attribute P1 in two data records A and B, the lower the character similarity of the attribute information corresponding to attribute P1 in two data records A and B.

[0069] Furthermore, if the number of different numbers for attribute P1 in two data records A and B is the set first threshold, the weight of the influence corresponding to attribute P1 is determined as the set fourth weight value; if the number of different numbers for attribute P1 in two data records A and B is the set second threshold, the weight of the influence corresponding to attribute P1 is determined as the set fifth weight value. Wherein, the first threshold < the second threshold; the fourth weight value > the fifth weight value. For example, if the first threshold is 1, the fourth weight value is 0.5; if the second threshold is 2, the fourth weight value is 0.3, etc.

[0070] Accordingly, if the number of different values ​​for attribute P1 in two data records A and B is greater than the set second threshold, the weight of the influence corresponding to attribute P1 is determined to be the set sixth weight value. The sixth weight value can be 0 or any other value less than the fifth weight value.

[0071] In other embodiments, if the attribute information of any attribute P1 is not purely numeric, determining the character similarity of the attribute information corresponding to attribute P1 in two data records A and B based on the attribute information corresponding to attribute P1 in two data records A and B can be achieved as follows: based on the attribute information corresponding to attribute P1 in any two data records A and B, determine the number of identical characters in the attribute information corresponding to attribute P1 in two data records A and B; based on the number of identical characters in the attribute information corresponding to attribute P1 in two data records A and B, determine the character similarity of the attribute information corresponding to attribute P1 in two data records A and B.

[0072] Optionally, based on the attribute information corresponding to attribute P1 in any two data records A and B, the longest common subsequence of attribute information corresponding to attribute P1 in the two data records can be determined; the number of characters in the longest common subsequence can be counted as the number of identical characters in the attribute information corresponding to attribute P1 in the two data records A and B.

[0073] Furthermore, the character similarity of attribute information corresponding to attribute P1 in two data records A and B can be calculated based on the number of characters in the longest common subsequence corresponding to any attribute P1 in both data records A and B, and the total number of characters in the non-purely numeric attribute information corresponding to attribute P1 in both data records A and B. Optionally, the character similarity of attribute information corresponding to attribute P1 in two data records A and B can be expressed as: Where x represents the number of characters in the longest common subsequence; 2 represents 2 data records; X and Y represent the number of characters in the two data records A and B, respectively.

[0074] Furthermore, the character similarity of attribute information corresponding to attribute P1 in two data records A and B can be used as the weight of the influence of attribute P1.

[0075] The above embodiments illustrating the implementation of determining the weights of the influence of attributes are merely illustrative and do not constitute a limitation.

[0076] After determining the weight of the influence corresponding to an attribute, the similarity between any two data records and their corresponding data objects in any attribute P1 can be determined based on the influence of any attribute P1 and its weight. For example, the product of the influence of any attribute P1 and its weight can be used as the similarity between any two data records and their corresponding data objects in any attribute P1.

[0077] Following the implementation methods described in the above embodiments for determining the similarity of data objects corresponding to two data records in any attribute, the similarity of data objects corresponding to two data records in each attribute can be determined. Then, based on the similarity of data objects corresponding to any two data records A and B in each attribute, the similarity between the data objects corresponding to data records A and B can be determined.

[0078] For data object identifiers, the similarity of the data objects corresponding to two data records A and B in terms of data object identifiers can be used as the similarity between the data objects corresponding to data records A and B.

[0079] For attributes other than the data object identifier, the sum of the similarities of the data objects corresponding to any two data records A and B in their other attributes can be calculated as the similarity between the data objects corresponding to data records A and B. In the same way, the similarity between the data objects corresponding to multiple data records in step 101 can be determined.

[0080] Furthermore, in step 103, data records belonging to the same data object can be determined from the multiple data records based on the similarity between the data objects corresponding to the multiple data records. Optionally, data objects with a similarity greater than or equal to a set similarity threshold can be selected from the data objects corresponding to the multiple data records as data records belonging to the same data object.

[0081] In the above implementation method of first clustering multiple data records to obtain at least one cluster, step 103 can be implemented as follows: For any cluster Q, based on the similarity between the data objects corresponding to the data records in cluster Q, data objects with a similarity greater than or equal to a set similarity threshold are selected from the data records contained in cluster Q as data records belonging to the same data object. Using the same method, data records belonging to the same data object can be determined from each cluster.

[0082] After identifying the data records belonging to the same data object, in step 104, an association network for the data object can be constructed based on the data records belonging to the same data object.

[0083] In some embodiments, such as Figure 3 As shown, each data record can be treated as a node, and edges can be added to data records with a similarity greater than or equal to a set similarity threshold. The weight of the edge is determined by the similarity between the data objects corresponding to the two nodes connected by the edge, thus constructing at least one connected subgraph (corresponding to...). Figure 2 Constructing connected subgraphs), such as Figure 3 Subgraphs (2) and (3) are shown. Each connected subgraph corresponds to a data object.

[0084] exist Figure 3 In the diagram, subgraph (1) represents the nodes corresponding to multiple data records in step 101; each node represents a data record. Using the clustering process shown in the above embodiment for data records within the same cluster, the cluster shown in subgraph (2) is obtained. Figure 3 In the subgraph (2), nodes that are not connected to each other belong to different clusters.

[0085] Furthermore, the similarity between data objects corresponding to data records in the same cluster is calculated using the method shown in the above embodiments. Figure 3 In the subgraph (2), the similarity between data objects corresponding to nodes connected by solid lines is greater than or equal to the set similarity threshold; the similarity between data objects corresponding to nodes connected by dashed lines is less than the set similarity threshold.

[0086] Furthermore, by determining the data records belonging to the same data object based on the similarity between data records corresponding to data objects within the same cluster, we can obtain... Figure 3 Subgraph (3). Connected nodes in subgraph (3) represent data records belonging to the same data object, forming a connected subgraph of that data object. Unconnected nodes in subgraph (3) represent data records that do not belong to the same data object. Figure 3 In the subgraph (3), each node represents a data record belonging to the same data object, and the connection between any two nodes represents the association between the data records corresponding to the two nodes. For two nodes connected by multiple edges, the edge with the largest weight can be retained according to the weight of the multiple edges.

[0087] In some embodiments, a connected subgraph of the same data object can be used as an association network for that data object. In other embodiments, to facilitate subsequent maintenance or retrieval of the association network, a unique identifier can be constructed for the same data object. Based on this, the completeness of the attribute information contained in each node of the connected subgraph of the same data object can be determined according to the validity of the attribute information contained in each node. For the implementation method of determining the validity of the attribute information contained in the data record corresponding to the node, please refer to the relevant content of the above embodiments, which will not be repeated here.

[0088] In some embodiments, validity scores for each attribute of a data object can be preset. If the attribute information of any attribute P1 contained in a data record is valid, then the validity score of attribute P1 is determined to be the preset validity score of attribute P1; if the attribute information of any attribute P1 contained in a data record is invalid, then the validity score of attribute P1 is determined to be 0. Further, the sum of the validity scores of each attribute contained in a data record can be calculated as the completeness of the attribute information contained in the nodes of the data record.

[0089] Furthermore, the primary index node (corresponding to) of the connected subgraph can be determined based on the completeness of the attribute information contained in each node. Figure 2 Within a connected subgraph, the primary index node can be selected. For example, the node with the highest completeness of attribute information among all nodes in the connected subgraph can be chosen as the primary index node. Figure 3 In the subgraph (4), the black nodes are the main index nodes of the connected subgraph.

[0090] Furthermore, all nodes in the connected subgraph except the primary index node can be made contiguous with the primary index node; and all other connections between the nodes in the connected subgraph except those with the primary index node can be deleted to obtain the target connected subgraph associated with the primary index node, which serves as the association network for the same data object (corresponding to...). Figure 2 (Settings are performed within the connected subgraph). For example... Figure 3 In the subgraph (4), all other nodes in the connected subgraph are connected to the main index node.

[0091] In this embodiment, the similarity between data objects corresponding to multiple data records is determined based on the influence of data object attributes on the identification results of the same data object, and the attribute information of each data object. Then, based on the similarity between the data objects corresponding to multiple data records, data records belonging to the same data object are identified, thus achieving the identification of data records belonging to the same object. Since the influence of data object attributes on the identification results of the same data object is considered when determining the similarity between data objects corresponding to multiple data records, it helps to improve the accuracy of subsequent identification of the same data object. On the other hand, when identifying data records belonging to the same data object, the comprehensive consideration of various attributes of the data object helps to improve the completeness and comprehensiveness of the data record collection of the same data object, thereby helping to improve the comprehensiveness and completeness of the subsequently constructed association network of data objects.

[0092] In practical applications, data objects may generate new data records, and new data objects may also appear. For example, in a product maintenance scenario, a product may have new maintenance records, and new products may also have maintenance records. In a medical scenario, a patient may have new medical records, and new patients may have medical records. In this embodiment, for newly added data records, the similarity between the data objects corresponding to the newly added data records and the similarity between the data objects corresponding to the newly added data records and the data objects corresponding to the nodes in the target connected subgraph obtained in the above embodiments can be determined based on the attribute information of the data objects contained in the newly added data records and the attribute information contained in each node in the target connected subgraph. For specific implementation methods, please refer to the relevant content of the above embodiments, which will not be repeated here.

[0093] In this embodiment, it is only necessary to calculate the similarity between the data objects corresponding to the newly added data record and the similarity between the data object corresponding to the newly added data record and the data objects corresponding to each node in the target connected subgraph. There is no need to repeatedly calculate the similarity between each node in the target connected subgraph, which can reduce the amount of similarity calculation.

[0094] Furthermore, based on the similarity between the data objects corresponding to the newly added data records and the similarity between the data objects corresponding to the newly added data records and the data objects corresponding to each node in the target connected subgraph, the target newly added data records that belong to the same data object as the target connected subgraph can be determined from the newly added data records. For the specific implementation method, please refer to the relevant content of step 103 above, which will not be repeated here.

[0095] For newly added data records that do not belong to the same data object as the target connected subgraph, the newly added data records that belong to the same data object can be determined based on the similarity between the data objects corresponding to these newly added data records. For specific implementation methods, please refer to the relevant content of step 103 and its specific implementation methods above, which will not be repeated here.

[0096] For example, such as Figure 4 As shown, circles represent data records corresponding to nodes in the target connected subgraph, and squares represent newly added data records. Through the above implementation method, the target newly added data records belonging to the same data object as the target connected subgraph and the newly added data records belonging to the same data object can be determined. Among them, Figure 4 The newly added data records corresponding to the square nodes connected to the circle in subgraph (2) are the target newly added data records belonging to the same data object as the target connected subgraph; the newly added data records between the square nodes that are connected to each other and not connected to the circle nodes are the newly added data records belonging to the same data object.

[0097] For nodes connected by multiple edges, the edges with the highest similarity can be retained and the others deleted based on the similarity between the data objects corresponding to the connected nodes. Figure 4 Subgraph (3).

[0098] For embodiments where a newly added data record belongs to the same data object as the target connected subgraph, the target connected subgraph of the same data object can be updated based on the newly added data record. Optionally, the target primary index node can be determined based on the completeness of the attribute information contained in the primary index node of the target connected subgraph and the completeness of the attribute information contained in the newly added data record. For specific implementation details regarding the target primary index node, please refer to the relevant content of the above embodiments, which will not be repeated here. The target primary index node can be the node corresponding to the newly added data record or the original primary index node of the target connected subgraph, specifically determined by the completeness of the attribute information contained in each node.

[0099] After identifying the target primary index node, the target connected subgraph for the same data object can be updated based on it. Specifically, the target connected subgraph and the nodes corresponding to the newly added data records can be connected to the target primary index, while edges between other nodes can be deleted, resulting in... Figure 4 Neutron diagram (4). In Figure 4 In the middle, the black nodes are the main index nodes, and the gray nodes are other nodes.

[0100] The target primary index node can be the node corresponding to a newly added data record in the target, or it can be the original primary index node of the target connected subgraph, determined by the completeness of the attribute information contained in each node. For example, Figure 5 As shown in subgraphs (2) and (4), the target primary index node is the original primary index node; while Figure 6 The main index node in subgraphs (1)-(3) is the node corresponding to the newly added data record in the target.

[0101] It should be noted that, Figure 5 Independent connected subgraphs composed of square nodes (e.g.) Figure 5 The nodes in subgraphs (1) and (5) are newly added data records that do not belong to the same data object as the original target connected subgraph. For the specific implementation of determining the main index node of the connected subgraph corresponding to the newly added data record that does not belong to the same data object as the existing target connected subgraph, please refer to the relevant content of the above embodiments, which will not be repeated here.

[0102] In practical applications, there may be situations where a newly added data record of the target is associated with multiple target connected subgraphs, meaning the similarity between the newly added data record of the target and data records in multiple target connected subgraphs exceeds a set similarity threshold. For example... Figure 5Neutron diagram (6) and Figure 6 As shown in the subgraph (4), the newly added data records of the target can be associated with the main index nodes of multiple target connected subgraphs respectively to obtain the connected graph of the multiple target connected subgraphs.

[0103] Furthermore, the target primary index node of the connected graph of multiple target connected subgraphs can be determined based on the completeness of the attribute information contained in the primary index node of the multiple target connected subgraphs and the completeness of the attribute information contained in the newly added target data record. The method for determining the target primary index node can be found in the relevant content of the above embodiments, and will not be repeated here. Furthermore, the connected graph of multiple target connected subgraphs can be updated based on the target primary index node to update the target connected subgraph of the same data object. Specifically, edges between each node and the target primary index node can be established in the connected graph of multiple target connected subgraphs, and connections between each node and other nodes except the target primary index node can be deleted, resulting in the following: Figure 5 Neutron diagram (6) and Figure 6 The connected graph shown in the lower part of the subgraph (4) serves as the target connected subgraph for the same data object. Figure 5 The lower part of the subgraph (6) shows that the target primary index node is the original primary index node; Figure 6 The lower part of the subgraph (4) shows that the target primary index node is the node corresponding to the newly added data record.

[0104] The updating method of the association network of data objects shown in the above embodiments can realize the timely updating of the knowledge base of data objects, which helps to further improve the comprehensiveness and completeness of the knowledge base of data objects.

[0105] The association network of the same data object obtained in the above embodiments has different functions in different application scenarios. For example, in a product operation and maintenance scenario, the operation and maintenance association network of the same product can be obtained by using the data processing method provided in the above embodiments based on the operation and maintenance records of the same product. This operation and maintenance association network can provide operation and maintenance personnel with operation and maintenance references for the product. Of course, this operation and maintenance association network can also provide a reference for R&D personnel to improve the product.

[0106] For example, in a medical setting, based on a patient's medical records, the data processing methods provided in the above embodiments can be used to obtain more comprehensive data about the patient. Therefore, more comprehensive health record data can be constructed, and the associated network can facilitate doctors to have a more comprehensive understanding of the patient's health status and provide more comprehensive and complete treatment plans, etc.

[0107] For example, in an internet context, user behavior data can be used to obtain a user behavior habit association network through the data processing methods provided in the above embodiments. Furthermore, based on this network, content matching the user's behavior habits, such as articles, products, videos, or games, can be recommended to the user.

[0108] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 and 102 can be device A; or the execution subject of step 101 can be device A, and the execution subject of step 102 can be device B; and so on.

[0109] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0110] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause one or more processors to perform the steps in the above-described data processing method.

[0111] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 7 As shown, the computing device mainly includes a memory 70a and a processor 70b. The memory 70a is used to store computer programs.

[0112] The processor 70b is coupled to the memory 70a and is used to execute a computer program for: acquiring multiple data records; the multiple data records include attribute information of data objects; determining the similarity between the data objects corresponding to the multiple data records based on the attribute information of the data objects contained in the multiple data records and the influence degree corresponding to each attribute of the data objects; and determining the data records belonging to the same data object from the multiple data records based on the similarity between the data objects corresponding to the multiple data records; and then constructing an association network of the same data object based on the data records of the same data object.

[0113] Optionally, when determining the similarity between data objects corresponding to multiple data records, the processor 70b specifically performs the following steps: obtaining key attribute information corresponding to each of the multiple data records from the attribute information of the data objects contained in the multiple data records; clustering the multiple data records according to the key attribute information corresponding to each of the multiple data records to obtain at least one cluster; and for any cluster, calculating the similarity between data objects corresponding to data records in any cluster based on the attribute information of the data objects contained in the data records in any cluster and the influence degree of each attribute of the data objects.

[0114] Furthermore, when clustering multiple data records based on their respective key attribute information, the processor 70b specifically performs the following: For any key attribute information corresponding to any data record among the multiple data records, it performs validity checks on the key attribute information corresponding to that data record to determine the valid key attribute information corresponding to that data record; based on the fault tolerance method corresponding to the key attributes, it performs fault tolerance processing on the valid key attribute information corresponding to the multiple data records to obtain the fault-tolerant valid key attribute information corresponding to the multiple data records; and based on the fault-tolerant valid key attribute information corresponding to the multiple data records, it clusters the multiple data records.

[0115] Optionally, when the processor 70b performs validity checks on the key attribute information corresponding to any data record, it specifically performs validity checks on the key attribute information corresponding to any data record based on the number of valid bits of the key attribute and the data organization rules, so as to determine the valid key attribute information corresponding to any data record.

[0116] Furthermore, when processor 70b performs validity checks on key attribute information corresponding to any data record based on the number of valid bits of the key attribute and data organization rules, it specifically performs the following steps: according to the data organization rules of the key attribute, it deletes characters that do not conform to the data organization rules from the key attribute information corresponding to any data record to obtain the valid bits of the key attribute information corresponding to any data record; if the number of valid bits of the key attribute information corresponding to any data record is greater than or equal to the number of valid bits of the corresponding key attribute, it determines that the key attribute information corresponding to any data record is valid key attribute information; or, if the number of valid bits of the key attribute information corresponding to any data record is greater than or equal to the number of valid bits of the corresponding key attribute, it performs validity checks on the valid bits of the key attribute information corresponding to any data record according to the data organization rules of the key attribute to determine the validity of the valid bits of the key attribute information corresponding to any data record.

[0117] In other embodiments, when processor 70b performs fault-tolerance processing on valid key attribute information corresponding to multiple data records according to the fault-tolerance method corresponding to the key attribute, it is specifically used for: Based on the data organization rules of key attributes, data supplementation is performed on the effective key attribute information corresponding to multiple data records to obtain effective key information that conforms to the data organization rules, which serves as the effective key attribute information after fault tolerance processing for multiple data records. And / or, Convert the valid key attribute information corresponding to multiple data records into phonetic symbols; remove the phonetic symbols set in the phonetic symbols of the valid key attribute information of multiple data records to obtain the valid phonetic symbols of the valid key attribute information of multiple data records, which are used as the valid key attribute information after fault tolerance processing corresponding to multiple data records.

[0118] Optionally, when the processor 70b clusters multiple data records based on the valid key attribute information after fault tolerance processing corresponding to the multiple data records, it specifically performs the following: based on the valid key attribute information after fault tolerance processing corresponding to the multiple data records, it clusters the data records with the same valid key attribute information after fault tolerance processing into a cluster.

[0119] In some embodiments of this application, when the processor 70b calculates the similarity between data objects corresponding to data records in any cluster, it is specifically used to: for any two data records in any cluster, determine the similarity of the data objects corresponding to the first attribute based on the attribute information and the influence degree corresponding to the first attribute in the two data records; determine the similarity between the data objects corresponding to the two data records based on the similarity of the data objects corresponding to the two data records in each attribute; wherein, the first data is any attribute of the data object.

[0120] Optionally, when processor 70b calculates the similarity of data objects corresponding to any two data records in the first attribute based on the attribute information corresponding to the first attribute in any two data records and the influence degree corresponding to the first attribute, it specifically performs the following: performs validity detection on the attribute information corresponding to the first attribute in any two data records to determine the validity of the first attribute in any two data records; and determines the similarity of data objects corresponding to any two data records in the first attribute based on the validity of the first attribute in any two data records and the influence degree of the first attribute on the corresponding data.

[0121] In some embodiments, the first attribute is a data object identifier. Accordingly, when the processor 70b determines the similarity of the data objects corresponding to any two data records in the first attribute based on the validity of the first attribute in any two data records and the influence of the first attribute, it specifically performs the following: if the data object identifiers in both data records are valid, it determines whether the data object identifiers in the two data records are the same; if the determination result is yes, it uses the influence of the data object identifier on the identification result of the same data object as the similarity of the data objects corresponding to the two data records in the data object identifier; if the data object identifier in any two data records is invalid, or if the data object identifiers in any two data records are different, it determines the similarity of the data objects corresponding to the two data records in the first attribute based on the validity of other attributes in the two data records besides the data object identifier and the influence of other attributes on the identification result of the same data object.

[0122] In other embodiments, the first attribute is an attribute other than the data object identifier. Accordingly, when the processor 70b determines the similarity of the data objects corresponding to any two data records in the first attribute based on the validity of the first attribute in any two data records and the influence degree corresponding to the first attribute, it specifically performs the following: if there are invalid cases of the first attribute in any two data records, it determines the weight of the influence degree corresponding to the first attribute based on the number of valid first attributes in any two data records; and it determines the similarity of the data objects corresponding to any two data records in the first attribute based on the influence degree corresponding to the first attribute and the weight of the influence degree corresponding to the first attribute.

[0123] In other embodiments, the first attribute is an attribute other than the data object identifier. When the processor 70b determines the similarity of data objects corresponding to any two data records in the first attribute based on the validity of the first attribute in any two data records and the influence degree corresponding to the first attribute, it specifically performs the following: if the first attribute is valid in both data records, it determines the character similarity of the attribute information corresponding to the first attribute in any two data records based on the attribute information corresponding to the first attribute in any two data records; it determines the weight of the influence degree of the first attribute on the identification result of the same data object based on the character similarity of the attribute information corresponding to the first attribute in any two data records; and it determines the similarity of data objects corresponding to any two data records in the first attribute based on the influence degree corresponding to the first attribute and the weight corresponding to the first attribute.

[0124] Optionally, the attribute information of the first attribute is purely numeric. Accordingly, when determining the character similarity of the attribute information corresponding to the first attribute in any two data records based on the attribute information corresponding to the first attribute in those two data records, the processor 70b specifically performs the following: sequentially compares the purely numeric attribute information corresponding to the first attribute in any two data records to obtain the number of different numbers in the first attribute; and determines the character similarity of the attribute information corresponding to the first attribute in any two data records based on the number of different numbers in the first attribute.

[0125] Optionally, the attribute information of the first attribute is non-pure numeric attribute information. Accordingly, when determining the character similarity of the attribute information corresponding to the first attribute in any two data records based on the attribute information corresponding to the first attribute in those two data records, the processor 70b specifically performs the following steps: Based on the non-pure numeric attribute information corresponding to the first attribute in any two data records, determine the longest common subsequence of the non-pure numeric attribute information corresponding to the first attribute in those two data records; based on the number of characters in the longest common subsequence of the first attribute in those two data records and the total number of characters in the non-pure numeric attribute information corresponding to the first attribute in those two data records, determine the similarity of the longest common subsequence of the first attribute in those two data records as the character similarity of the attribute information corresponding to the first attribute in those two data records.

[0126] Accordingly, when determining the weight of the influence degree corresponding to the first attribute based on the character similarity of the attribute information corresponding to the first attribute in any two data records, the processor 70b is specifically used to: take the longest common subsequence similarity corresponding to the first attribute in the two data records as the weight of the influence degree corresponding to the first attribute.

[0127] In this embodiment of the application, when the processor 70b constructs an association network for the same data object based on data records of the same data object, it is specifically used to: take each data record as a node, add edges between nodes corresponding to data records with similarity greater than or equal to a set similarity threshold, and use the similarity between the data objects corresponding to the two nodes connected by the edge as the weight of the edge to construct at least one connected subgraph; each connected subgraph corresponds to a data object; and construct an association network for the data object based on the connected subgraph of any data object.

[0128] Optionally, when the processor 70b constructs the association network of any data object based on the connected subgraph of any data object, it specifically performs the following steps: determining the completeness of the attribute information contained in each node of the connected subgraph of any data object based on the validity of the attribute information contained in each node; determining the main index node of the connected subgraph based on the completeness of the attribute information contained in each node of the connected subgraph; connecting other nodes in the connected subgraph to the main index node; and deleting other connections in the connected subgraph except for those to the main index node, to obtain the target connected subgraph associated with the main index node, which serves as the association network of the data object.

[0129] Optionally, the processor 70b is further configured to: acquire newly added data records; determine the similarity between data objects corresponding to the newly added data records and the similarity between data objects corresponding to the newly added data records and data objects corresponding to nodes in the existing target connected subgraph, based on the attribute information of the data objects contained in the newly added data records and the attribute information contained in each node in the existing target connected subgraph; determine the target newly added data records that belong to the same data object as the target connected subgraph from the newly added data records, based on the similarity between data objects corresponding to the newly added data records and the similarity between data objects corresponding to the newly added data records and data objects corresponding to each node in the existing target connected subgraph; and update the target connected subgraph of the same data object based on the target newly added data records.

[0130] Optionally, when updating the target connected subgraph of the same data object based on the target newly added data record, the processor 70b specifically performs the following: determining the target main index node based on the completeness of the attribute information contained in the main index node of the target connected subgraph and the completeness of the attribute information contained in the target newly added data record; and updating the target connected subgraph of the same data object based on the target main index node.

[0131] In some embodiments, the target newly added data record is associated with multiple existing target connected subgraphs. When updating the target connected subgraph of the same data object based on the target newly added data record, the processor 70b is specifically configured to: associate the target newly added data record with the main index nodes of the multiple target connected subgraphs respectively to obtain a connected graph of the multiple target connected subgraphs; determine the target main index node of the connected graph of the multiple target connected subgraphs based on the completeness of the attribute information contained in the main index nodes of the multiple target connected subgraphs and the completeness of the attribute information contained in the target newly added data record; and update the connected graph of the multiple target connected subgraphs based on the target main index node of the connected graph of the multiple target connected subgraphs to update the target connected subgraph of the same data object.

[0132] In some alternative implementations, such as Figure 7As shown, the computing device may also include components such as a communication component 70c, a power supply component 70d, a display component 70e, and an audio component 70f. Figure 7 The diagram only shows some components and does not mean that the computing device must contain them. Figure 7 The inclusion of all components does not imply that a computing device can only include... Figure 7 The components shown.

[0133] The computing device provided in this embodiment, in order to construct an association network of data objects, can determine the similarity between data objects corresponding to multiple data records based on the influence of each attribute of the data object on the identification result of the same data object, and the attribute information of the data object; and based on the similarity between the data objects corresponding to multiple data records, determine the data records belonging to the same data object, thus realizing the identification of data records of the same object. Since the influence of data object attributes on the identification result of the same data object is considered when determining the similarity between data objects corresponding to multiple data records, it helps to improve the accuracy of subsequent identification of the same data object. On the other hand, when identifying data records of the same data object, the comprehensive consideration of various attributes of the data object helps to improve the completeness and comprehensiveness of the collection of data records of the same data object, thereby helping to improve the comprehensiveness and completeness of the association network of the data objects subsequently constructed.

[0134] In this embodiment, the memory is used to store computer programs and can be configured to store various other data to support operation on its host device. The processor can execute the computer programs stored in the memory to implement corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0135] In the embodiments of this application, the processor can be any hardware processing device capable of executing the above-described method logic. Optionally, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), or a microcontroller unit (MCU); it can also be a programmable device such as a field-programmable gate array (FPGA), a programmable array logic (PAL), a general array logic (GAL), or a complex programmable logic device (CPLD); or it can be an advanced reduced instruction set (RISC) processor (ARM) or a system on chip (SOC), etc., but is not limited thereto.

[0136] In this embodiment, the communication component is configured to facilitate wired or wireless communication between its host device and other devices. The device housing the communication component can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In another exemplary embodiment, the communication component may also be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), Bluetooth (BT), or other technologies.

[0137] In embodiments of this application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component may be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation.

[0138] In this embodiment, a power supply component is configured to provide power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

[0139] In embodiments of this application, the audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), which is configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals. For example, in devices with voice interaction capabilities, voice interaction with the user can be achieved through the audio component.

[0140] It should be noted that the terms "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0141] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0147] Computer storage media are readable storage media, also known as removable media. Removable and non-removable media can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transient media, such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data processing method, characterized in that, include: Retrieve multiple data records; the data records include attribute information of data objects; From the attribute information of the data objects contained in the multiple data records, obtain the key attribute information corresponding to each of the multiple data records; Based on the number of valid bits of the key attributes and the data organization rules, the key attribute information corresponding to any one of the multiple data records is tested for validity in order to determine the valid key attribute information corresponding to the data record. Based on the fault tolerance method corresponding to the key attributes, fault tolerance processing is performed on the valid key attribute information corresponding to the multiple data records to obtain the fault-tolerant valid key attribute information corresponding to the multiple data records; the fault tolerance processing of the valid key attribute information corresponding to the multiple data records includes: converting the valid key attribute information corresponding to the multiple data records into phonetic symbols; removing the set phonetic symbols from the phonetic symbols of the valid key attribute information of the multiple data records to obtain the valid phonetic symbols of the valid key attribute information of the multiple data records, which serve as the fault-tolerant valid key attribute information corresponding to the multiple data records; wherein, removing the set phonetic symbols from the phonetic symbols of the valid key attribute information of the multiple data records includes: removing the retroflex consonant symbols and the nasal consonant symbols from the phonetic symbols of the valid key attribute information of the multiple data records; Data records that have the same valid key attribute information after fault tolerance processing are clustered into a single cluster. For any cluster, the similarity between data objects corresponding to data records in any cluster is calculated based on the attribute information of the data objects contained in the data records in the cluster and the influence degree of each attribute of the data objects. Based on the similarity between the data objects corresponding to the multiple data records, determine the data records belonging to the same data object from the multiple data records; Based on the data records of the same data object, construct the association network of the same data object; The step of constructing the association network of the same data object based on the data records of the same data object includes: Each data record is treated as a node. Edges are added between nodes corresponding to data records whose similarity is greater than or equal to a set similarity threshold. The weight of the edge is the similarity between the data objects corresponding to the two nodes connected by the edge. At least one connected subgraph is constructed. Each connected subgraph corresponds to one data object. The completeness of the attribute information contained in each node of the connected subgraph of any data object is determined based on the validity of the attribute information contained in each node of the connected subgraph of any data object. The main index node of the connected subgraph is determined based on the completeness of the attribute information contained in each node of the connected subgraph of any data object. Connect other nodes in the connected subgraph of any data object to the main index node; and delete the other connections in the connected subgraph of any data object except those to the main index node, to obtain the target connected subgraph associated with the main index node, which serves as the association network of any data object. The step of determining the completeness of the attribute information contained in each node of the connected subgraph of any data object based on the validity of the attribute information contained in each node of the connected subgraph of any data object includes: For any node in the connected subgraph of any data object, if any attribute information contained in any node is valid attribute information, then the validity score of any attribute information is determined to be the preset validity score of any attribute information; if any attribute information is invalid, then the validity score of any attribute information is determined to be 0. The sum of the validity scores of each attribute information contained in any given node is calculated as the completeness of the attribute information contained in any given node.

2. The method according to claim 1, characterized in that, The step of performing fault tolerance processing on the valid key attribute information corresponding to the multiple data records according to the fault tolerance method corresponding to the key attributes also includes: According to the data organization rules of the key attributes, the valid key attribute information corresponding to the multiple data records is supplemented to obtain valid key information that conforms to the data organization rules, which serves as the valid key attribute information after fault tolerance processing for the multiple data records.

3. The method according to claim 1, characterized in that, The step of calculating the similarity between data objects corresponding to data records in any cluster based on the attribute information of data objects contained in data records in any cluster, and the influence degree corresponding to each attribute of the data object, includes: For any two data records in any cluster, the similarity of the data objects corresponding to the first attribute is determined based on the attribute information corresponding to the first attribute in the two data records and the influence degree corresponding to the first attribute; the first attribute is any attribute of the data object. The similarity between the data objects corresponding to any two data records is determined based on the similarity of their attributes.

4. The method according to claim 3, characterized in that, The step of calculating the similarity of the data objects corresponding to the first attribute between any two data records based on the attribute information corresponding to the first attribute and the influence degree corresponding to the first attribute includes: The validity of the attribute information corresponding to the first attribute in any two data records is determined by performing a validity check on the first attribute in any two data records. Based on the validity of the first attribute in any two data records and the influence degree corresponding to the first attribute, determine the similarity of the data objects corresponding to the two data records in the first attribute.

5. The method according to claim 4, characterized in that, The first attribute is a data object identifier; determining the similarity of the data objects corresponding to any two data records in the first attribute based on the validity of the first attribute in any two data records and the influence degree corresponding to the first attribute includes: If the data object identifiers in any two data records are valid, determine whether the data object identifiers in any two data records are the same; if the determination result is yes, use the influence degree corresponding to the data object identifier as the similarity of the data objects corresponding to the two data records in the data object identifier. If any two data records contain invalid data object identifiers, or if any two data records contain different data object identifiers, the similarity of the data objects corresponding to the two data records in the first attribute is determined based on the validity of other attributes in the two data records besides the data object identifiers and the influence of the other attributes.

6. The method according to claim 4, characterized in that, The first attribute is any attribute other than the data object identifier; determining the similarity of the data objects corresponding to any two data records in the first attribute based on the validity of the first attribute in the two data records and the influence degree corresponding to the first attribute includes: If any two data records contain invalid first attributes, the weight of the influence corresponding to the first attribute is determined based on the number of valid first attributes in the two data records; and the similarity of the data objects corresponding to the first attribute is determined based on the influence corresponding to the first attribute and the weight of the influence corresponding to the first attribute. If the first attribute is valid in any two data records, determine the character similarity of the attribute information corresponding to the first attribute in any two data records based on the attribute information corresponding to the first attribute in any two data records; determine the weight of the influence corresponding to the first attribute based on the character similarity of the attribute information corresponding to the first attribute in any two data records; determine the similarity of the data objects corresponding to the first attribute in any two data records based on the influence corresponding to the first attribute and the weight of the influence corresponding to the first attribute.

7. The method according to claim 6, characterized in that, The attribute information of the first attribute is non-purely numeric attribute information; the step of determining the character similarity of the attribute information corresponding to any attribute in any two data records based on the attribute information corresponding to the first attribute in any two data records includes: Based on the non-pure numeric attribute information corresponding to the first attribute in any two data records, determine the longest common subsequence of the non-pure numeric attribute information corresponding to the first attribute in any two data records; Based on the number of characters in the longest common subsequence corresponding to the first attribute in any two data records, and the total number of characters in the non-pure numeric attribute information corresponding to the first attribute in the two data records, the similarity of the longest common subsequence corresponding to the first attribute in the two data records is determined, and is used as the character similarity of the attribute information corresponding to the first attribute in any two data records. The step of determining the weight of the influence corresponding to the first attribute based on the character similarity of the attribute information corresponding to the first attribute in any two data records includes: The longest common subsequence similarity corresponding to the first attribute in the two data records is used as the weight of the influence corresponding to the first attribute.

8. The method according to any one of claims 1-7, characterized in that, Also includes: Retrieve newly added data records; Based on the attribute information of the data objects contained in the newly added data record and the attribute information contained in each node of the existing target connected subgraph, the similarity between the data objects corresponding to the newly added data record and the similarity between the data objects corresponding to the newly added data record and the data objects corresponding to each node of the existing target connected subgraph are determined. Based on the similarity between the data objects corresponding to the newly added data records, and the similarity between the data objects corresponding to the newly added data records and the data objects corresponding to each node in the existing target connected subgraph, target new data records that belong to the same data object as the existing target connected subgraph are determined from the newly added data records. Based on the newly added data record for the target, update the target connected subgraph of the same data object.

9. The method according to claim 8, characterized in that, The newly added target data record is associated with multiple existing target connected subgraphs; updating the target connected subgraph of the same data object based on the newly added target data record includes: The newly added data records of the target are associated with the main index nodes of the multiple target connected subgraphs respectively to obtain the connected graph of the multiple target connected subgraphs; Based on the completeness of the attribute information contained in the main index nodes of the multiple target connected subgraphs and the completeness of the attribute information contained in the target newly added data records, the target main index nodes of the connected graph of the multiple target connected subgraphs are determined. Based on the target primary index node of the connected graph of the multiple target connected subgraphs, update the connected graph of the multiple target connected subgraphs to update the target connected subgraph of the same data object.

10. A computing device, characterized in that, include: A memory and a processor; wherein the memory is used to store computer programs; The processor is coupled to the memory for executing the computer program to perform the steps of the method according to any one of claims 1-9.

11. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed by one or more processors, the one or more processors cause the processors to perform the steps of the method according to any one of claims 1-9.

Images