A multi-data-source-based data integration utilization method

Abstract

The present application relates to the technical field of multi-source heterogeneous data processing, and discloses a data integration and utilization method based on multiple data sources, which comprises the following steps: obtaining multi-source business records, determining certainty identifiers, record time points and entity association records; records with the same certainty identifier are classified into the same entity record group; a cross-source field mapping relationship is determined according to the position relationship of the same value field; an entity association path is generated in combination with the entity association records; a candidate entity group is determined; the candidate entity group is incorporated into the entity record group and the field mapping relationship is updated; the entity association relationship of different data sources is continuously corrected; the attribute effective interval is determined according to the record time point in the entity alignment result; and a unified data view is generated, so that the interference of heterogeneous data is reduced, the stability of entity alignment is improved, and the consistency of data view is ensured.

Description

Technical Field

[0001] This invention relates to the field of multi-source heterogeneous data processing technology, and more specifically, to a method for integrating and utilizing data based on multiple data sources. Background Technology

[0002] Multi-source data integration and utilization methods are used in cross-system data processing scenarios for government departments and enterprises. The processing objects include database data, file data, and interface data. These methods are constrained by data format, encoding method, storage structure, and synchronization timeliness. Existing technologies typically use unified access, format standardization, quality verification, entity matching, time alignment, and attribute fusion to form a data view. These methods are usually suitable for multi-source data scenarios with stable data structure, complete entity identification, and gradual attribute changes.

[0003] During multi-source data integration, changes in the field structure of data sources can reduce the stability of field correspondences, and differences in the identification of the same entity in different systems can weaken the basis for entity association. These factors affect the generation of data views along the same processing chain of data standardization, entity matching, and attribute fusion, causing inconsistencies between entity records and attribute content in the data view. Therefore, the technical problem that needs to be solved is how to ensure the stability of data views while reducing heterogeneous interference and enhancing entity alignment stability. Figure 1 The issue of consistency.

[0004] In view of this, the present invention proposes a data integration and utilization method based on multiple data sources to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned shortcomings of the prior art, the present invention provides a data integration and utilization method based on multiple data sources.

[0006] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a method for integrating and utilizing data from multiple data sources is provided, including: Obtain multi-source business records, deterministic identifiers, record times, and entity association records, and group multi-source business records with the same deterministic identifier into the same entity record group; Determine the cross-source field mapping relationship based on the field position of fields with the same value and the position of adjacent fields within the entity record group; Based on the entity association record connection group, the entity association path is obtained by connecting the multi-source business records outside the entity association record group with the entity record group. The candidate entity group is determined by combining the entity association path and the cross-source field mapping relationship. The candidate entity group is merged into the entity record group. The cross-source field mapping relationship is updated according to the merged entity record group until the entity record group no longer increases, and the entity alignment result is obtained. The attribute effective range is determined according to the record time of the same name but different value fields in the entity alignment result. A unified data view is generated based on the entity alignment result and the attribute effective range.

[0007] In some embodiments, determining cross-source field mapping relationships based on the field positions of fields with the same value within an entity record group and the positions of adjacent fields includes: Read multi-source business records belonging to different data sources within an entity record group, and extract field values ​​based on field position; retain field records that have both field position and field value. Pair records with the same field value but different data sources; write the paired records into pairs of fields with the same value. Read the positions of the adjacent fields on both sides of the same value field pair to obtain the adjacent field position group; Write the field position of the same value field pair and the adjacent field position group to the field position record; associate the field position record with the corresponding entity record group; Determine cross-source field mapping relationships based on the field position records that appear repeatedly in different entity record groups.

[0008] In some embodiments, the positions of adjacent fields on both sides of a pair of fields with the same value are read to obtain a group of adjacent field positions, including: Based on the field positions of the same value field pairs in different data sources, read the first and second field positions respectively; write the read source to the first and second field positions; Read the field type corresponding to the preceding field position and the field type corresponding to the following field position; associate the field type with the corresponding preceding and following field positions; By combining the positions of the preceding and following fields that are of the same type and have the same relative direction, we obtain adjacent field position groups.

[0009] In some embodiments, the entity association path is obtained by connecting multi-source business records outside the entity association record group with the entity record group, including: Read multi-source business records that are not assigned to an entity record group to obtain the starting record; read the associated fields and record time in the starting record; Based on the associated fields in the starting record, retrieve the referenced record from the entity's associated records; then connect the referenced record and the starting record. Connect the starting record and the entity record group in the entity association record to obtain the forward record segment and the reverse record segment; Records with the same business records in the forward and reverse record segments but with opposite connection directions are taken as path intersection records, and the path intersection records are associated with the starting record. The entity association path is obtained by connecting the starting record, the path intersection record, and the entity record group in the order of their connection.

[0010] In some embodiments, the starting record and the entity record group are connected in the entity association record to obtain a forward record segment and a reverse record segment, including: Starting from the associated field of the starting record, read the referenced record with the same field value from the entity's associated record; write the connection relationship between the starting record and the referenced record into the forward path record; Starting from the multi-source business records within the entity record group, read the source records pointing to multi-source business records outside the group from the entity association records, and write the connection relationship between the multi-source business records and the source records within the entity record group into the reverse path record; Read the recording times from the forward path records and the reverse path records respectively; arrange the forward path records and the reverse path records according to the recording times; Based on the arranged forward path records and reverse path records, the forward record segment and the reverse record segment are obtained.

[0011] In some embodiments, writing the connection relationship between the starting record and the pointed-to record into the forward path record includes: Read the associated fields in the starting record to obtain the value of the starting associated field, and search for the record that is pointed to in the entity associated records with the same field value as the value of the starting associated field; Read the record time and field position of the pointed-to record, and write the record time and field position of the pointed-to record into the pointed-to record item; Write the starting record, the record being pointed to, and the connection direction between the starting record and the record being pointed to into the same record to obtain the forward path record.

[0012] In some embodiments, determining the attribute effective range based on the record time of identically named but different-valued fields within the entity alignment result includes: Read field records belonging to the same entity record group in the entity alignment result, and treat field records with the same field name but different field values ​​as field records with the same name but different values; Based on the business records corresponding to the same name but different values ​​field records, read the corresponding entity association path, and write the records of the same name but different values ​​field records into the corresponding entity association path at any time. Determine the effective range of attributes for records with the same name but different values ​​based on their position in the corresponding entity's association path at the time of the record.

[0013] In some embodiments, writing the record time of records with the same name but different values ​​into the corresponding entity association path includes: Read the record time of the corresponding business record in the same name but different value field, and read the record position of the corresponding business record in the entity association path in the same name but different value field. Read adjacent business records based on their record positions, and match the record times of adjacent business records with the record times of the corresponding business records with the same name but different values. Record the record time of the corresponding business record for the same name but different values ​​field and write it to the record location to obtain the corresponding entity association path at the record time.

[0014] In some embodiments, a unified data view is generated based on the entity alignment result and the attribute effective range, including: Read the entity record group from the entity alignment result and group the multi-source business records in the entity record group by field name; Read the attribute validity range and write the field name, field value, and attribute validity range into the corresponding entity record group; A unified data view is generated based on the entity record group whose field names, field values, and attribute validity ranges are written.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention acquires multi-source business records, deterministic identifiers, record times, and entity association records. Multi-source business records with the same deterministic identifier are grouped into the same entity record group, establishing initial associations between records from different sources. Cross-source field mapping relationships are determined based on the positional relationships of fields with the same value within the entity record group, reducing the impact of differences in field structure. Entity association paths are then formed by combining entity association records, and candidate entity groups are determined based on these paths and cross-source field mapping relationships. These candidate entity groups are merged into the entity record group, and the field mapping relationships are updated, enhancing entity alignment stability. Finally, a unified data view is generated based on the entity alignment results and attribute effective intervals, thereby reducing heterogeneous data interference and improving the consistency of multi-source data integration. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a data integration and utilization method based on multiple data sources in this invention. Figure 2 This is a schematic diagram of the structure of a data integration and utilization system based on multiple data sources in this invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. In the following detailed description, many specific details are set forth to provide a thorough understanding of the exemplary embodiments described. However, it will be apparent to those skilled in the art that the described embodiments may be practiced without some or all of these specific details. In other exemplary embodiments, well-known structures have not been described in detail to avoid unnecessarily obscuring the concepts of this disclosure. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention. Furthermore, the various aspects described in the embodiments may be combined arbitrarily without conflict.

[0018] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0019] Example 1

[0020] Figure 1 This disclosure illustrates at least one embodiment of a data integration and utilization method based on multiple data sources, comprising: S10: Obtain multi-source business records, deterministic identifiers, record times, and entity association records, and group multi-source business records with the same deterministic identifier into the same entity record group; In this embodiment, multi-source business records refer to data records from different business systems, databases, files, or interfaces. Deterministic identifiers refer to field content that can directly point to the same business entity. Record time refers to the time information corresponding to the generation, writing, or updating of multi-source business records. Entity association records refer to existing business connection relationships between different business records, such as the connection relationship between case records and party records, or the connection relationship between party records and execution records. Entity record groups refer to a set of multi-source business records that are classified into the same entity object.

[0021] Understandably, after obtaining multi-source business records, deterministic identifiers, record times, and entity association records, multi-source business records with the same deterministic identifier are first grouped into the same entity record group, thereby forming a basic entity scope with definite basis. For example, if the party record in the trial system and the execution record in the execution system both contain the same ID number, then the two multi-source business records are grouped into the same entity record group, providing a basic reference for subsequent cross-source field mapping relationships and entity association paths.

[0022] S20: Determine the cross-source field mapping relationship based on the field position of fields with the same value and the position of adjacent fields within the entity record group; In this embodiment, the same-value field refers to the field with the same value in multi-source business records from different data sources within an entity record group. Field position refers to the field's arrangement position in the corresponding multi-source business record. Adjacent field position refers to the field's arrangement position before or after the same-value field. Cross-source field mapping relationship refers to the correspondence between field position and business meaning between different data sources.

[0023] It should be noted that fields with the same business meaning may have different field names in different data sources. For example, one data source uses "party_id" and another uses "person_code". If the correspondence is established solely based on the field name, changes in the field structure will affect the stability of the field correspondence. Therefore, this embodiment reads the field position of the same value field and the positions of adjacent fields before and after the same value field based on the multi-source business records that have been confirmed to belong to the same entity in the entity record group. The same value field and adjacent fields jointly constrain the cross-source field mapping relationship.

[0024] Based on the field positions of fields with the same value and adjacent fields within the entity record group, determine the cross-source field mapping relationship, including: Read multi-source business records belonging to different data sources within an entity record group, and extract field values ​​based on field position; retain field records that have both field position and field value. Pair records with the same field value but different data sources; write the paired records into pairs of fields with the same value. Read the positions of the adjacent fields on both sides of the same value field pair to obtain the adjacent field position group; Write the field position of the same value field pair and the adjacent field position group to the field position record; associate the field position record with the corresponding entity record group; Determine cross-source field mapping relationships based on the field position records that appear repeatedly in different entity record groups.

[0025] In this embodiment, the field record includes field position and field value. The field position record includes the field position of the same value field pair, the adjacent field position groups on both sides of the same value field pair, and the corresponding entity record group. The field position record is not formed based on a single multi-source business record, but is formed among cross-source business records that have been grouped into the same entity record group.

[0026] Understandably, after reading multi-source business records belonging to different data sources within an entity record group, the field values ​​are first extracted according to the field position and the field records are retained. Then, field records with the same field value but different data sources are written into the same-value field pair. Subsequently, the adjacent field positions on both sides of the same-value field pair are read to form an adjacent field position group. The field position records are formed by the field positions of the same-value field pair and the adjacent field position groups, thus avoiding cross-source field mismatches caused solely by the same field value.

[0027] Furthermore, when the same field position records appear repeatedly in multiple entity record groups, it indicates that the corresponding field position and adjacent field positions maintain the same arrangement relationship in different entity objects. Therefore, the repeated field position records are used as the basis for the cross-source field mapping relationship. For example, if the ID number field value is the same in multiple party entity record groups, and the ID number field corresponds to the name field position and the contact information field position before and after it, then a cross-source field mapping relationship is established between the corresponding field positions.

[0028] Read the positions of the adjacent fields on both sides of the same value field pair to obtain the adjacent field position group, including: Based on the field positions of the same value field pairs in different data sources, read the first and second field positions respectively; write the read source to the first and second field positions; Read the field type corresponding to the preceding field position and the field type corresponding to the following field position; associate the field type with the corresponding preceding and following field positions; By combining the positions of the preceding and following fields that are of the same type and have the same relative direction, we obtain adjacent field position groups.

[0029] In this embodiment, the reading source refers to the data source to which the preceding and following field positions belong, the relative direction refers to the direction of the field position relative to the preceding or following direction of the same value field pair, and the field type refers to the data type or content type of the field value, such as text type, number type, date type, or amount type.

[0030] It should be noted that by reading the positions of the preceding and following fields based on the positions of the same value field pairs in different data sources, the field arrangement relationship can be preserved. Then, by writing the reading source into the preceding and following field positions, it is possible to distinguish which data source the adjacent field positions come from. Subsequently, the field type is read and associated with the adjacent field positions, so that the adjacent field position group simultaneously contains the position relationship, the source relationship, and the field type relationship.

[0031] It is understandable that if the first data source has a text field preceding the same value field and a date field following it, and the second data source also has a text field preceding the same value field and a date field following it, and the relative directions are the same, then the corresponding preceding and following field positions will be combined into an adjacent field position group. This makes the cross-source field mapping relationship not only dependent on the same value field itself, but also constrained by the arrangement of the surrounding fields of the same value field.

[0032] S30: Based on the entity association record connection group, the multi-source business records outside the entity association record group and the entity record group are obtained, and the candidate entity group is determined by combining the entity association path and the cross-source field mapping relationship; In this embodiment, "external multi-source business records" refers to multi-source business records that have not yet been assigned to any entity record group. "Entity association path" refers to the record connection relationship formed by external multi-source business records being connected to entity record groups through entity association records. "Candidate entity group" refers to the set of external multi-source business records that, after being constrained by entity association paths and cross-source field mapping relationships, have the basis to be incorporated into entity record groups.

[0033] Understandably, entity association records reflect existing business connections between business records, such as the connection between party records and case records, the connection between case records and execution records, and the connection between petition records and party records. After determining the entity association path, the field positions in different data sources can be matched with the cross-source field mapping relationship to avoid erroneous entity merging based solely on the connection of associated fields.

[0034] Furthermore, the entity association path is used to limit the association direction between multi-source business records outside the group and the entity record group, and the cross-source field mapping relationship is used to limit the comparable field positions between multi-source business records outside the group. Thus, a candidate range is first formed from the business connection relationship, and then the candidate entity group is confirmed from the field position relationship, so that the candidate entity group is continuously connected with the aforementioned entity record group and cross-source field mapping relationship.

[0035] Based on the entity association record connection group, the entity association path is obtained by linking the multi-source business records outside the entity association record group with the entity record group, including: Read multi-source business records that are not assigned to an entity record group to obtain the starting record; read the associated fields and record time in the starting record; Based on the associated fields in the starting record, retrieve the referenced record from the entity's associated records; then connect the referenced record and the starting record. Connect the starting record and the entity record group in the entity association record to obtain the forward record segment and the reverse record segment; Records with the same business records in the forward and reverse record segments but with opposite connection directions are taken as path intersection records, and the path intersection records are associated with the starting record. The entity association path is obtained by connecting the starting record, the path intersection record, and the entity record group in the order of their connection.

[0036] In this embodiment, the starting record is the record to be associated read from the multi-source business records outside the group, the associated field is the field in the starting record that points to other business records, the pointed-to record is the business record pointed to by the associated field in the entity associated record, the forward record segment is the record segment formed by extending from the starting record to the pointed-to record, the reverse record segment is the record segment formed by reverse searching from the entity record group to the multi-source business records outside the group, and the path intersection record is the business record that appears in both the forward record segment and the reverse record segment and has opposite connection directions.

[0037] It should be noted that reading the associated fields in the starting record in one direction is easily affected by differences in the data source field structure and the default of associated fields. Therefore, this embodiment simultaneously forms a forward record segment from the starting record and a reverse record segment from the entity record group. When the same business record appears in the forward record segment and the reverse record segment and the connection direction is opposite, the corresponding business record is used as the path intersection record, so that the connection relationship between the starting record and the entity record group has bidirectional reference.

[0038] Understandably, after obtaining the entity association path according to the connection order of the starting record, the path intersection record, and the entity record group, the entity association path includes the starting position of the multi-source business records outside the group, the intersection position that can be connected to the entity record group, and the position of the entity record group that has been grouped. This provides a traceable record connection basis for subsequently determining the candidate entity group by combining the cross-source field mapping relationship.

[0039] By concatenating the starting record and the entity record group in the entity association record, we obtain the forward record segment and the reverse record segment, including: Starting from the associated field of the starting record, read the referenced record with the same field value from the entity's associated record; write the connection relationship between the starting record and the referenced record into the forward path record; Starting from the multi-source business records within the entity record group, read the source records pointing to multi-source business records outside the group from the entity association records, and write the connection relationship between the multi-source business records and the source records within the entity record group into the reverse path record; Read the recording times from the forward path records and the reverse path records respectively; arrange the forward path records and the reverse path records according to the recording times; Based on the arranged forward path records and reverse path records, the forward record segment and the reverse record segment are obtained.

[0040] In this embodiment, the forward path record is used to store the connection relationship between the starting record and the pointed-to record, and the reverse path record is used to store the connection relationship between the multi-source business records within the entity record group and the source record. The source record refers to the record in the entity association record that points to the multi-source business record outside the group.

[0041] Furthermore, reading the associated field from the starting record can determine the record that the starting record points to, and reading the source record from the multi-source business record within the entity record group can determine the external pointing relationship of the entity record group. After the two types of connection relationships are written into the forward path record and the reverse path record respectively, they are arranged according to the record time, so that the forward record segment and the reverse record segment retain both the connection direction and the time order.

[0042] Write the connection relationship between the starting record and the record it points to into the forward path record, including: Read the associated fields in the starting record to obtain the value of the starting associated field, and search for the record that is pointed to in the entity associated records with the same field value as the value of the starting associated field; Read the record time and field position of the pointed-to record, and write the record time and field position of the pointed-to record into the pointed-to record item; Write the starting record, the record being pointed to, and the connection direction between the starting record and the record being pointed to into the same record to obtain the forward path record.

[0043] In this embodiment, the starting associated field value is the field value of the associated field in the starting record, the pointed-to record item is used to store the recording time and field position of the pointed-to record, and the connection direction is used to indicate the pointing relationship between the starting record and the pointed-to record.

[0044] It should be noted that when forming a forward path record, not only is the connection between the starting record and the pointed-to record recorded, but the recording time and field position of the pointed-to record are also written into the pointed-to record item. This allows the forward path record to express business connection, time position and field position at the same time. Subsequently, when forming path intersection records and candidate entity groups, it can be directly connected with entity association paths and cross-source field mapping relationships.

[0045] S40: Merge the candidate entity group into the entity record group, update the cross-source field mapping relationship according to the merged entity record group, until the entity record group no longer increases, and obtain the entity alignment result. Determine the attribute effective range according to the record time of the same name but different value fields in the entity alignment result, and generate a unified data view based on the entity alignment result and the attribute effective range.

[0046] In this embodiment, the candidate entity group is a set of multi-source business records outside the group determined by the entity association path and the cross-source field mapping relationship. The entity alignment result refers to the entity record group result formed after the candidate entity group is merged and the cross-source field mapping relationship is updated. The attribute effective interval refers to the effective time range of the corresponding field value of the same name but different value field in the unified data view. The unified data view refers to the data result organized according to the entity record group, field name, field value and attribute effective interval.

[0047] It should be noted that after merging the candidate entity group into the entity record group, the number of multi-source business records in the entity record group changes. The merged entity record group can provide new same-value fields and adjacent field positions. Therefore, it is necessary to update the cross-source field mapping relationship according to the merged entity record group. The updated cross-source field mapping relationship continues to participate in the candidate entity group determination until the entity record group no longer increases, thus obtaining the entity alignment result.

[0048] It is understandable that after obtaining the entity alignment results, there may still be fields with the same name but different values ​​within the same entity record group. For example, the contact information, address or status fields of the same party in different business systems may have different field values. In this case, instead of directly deleting one of the field values, the effective range of the attribute is determined according to the record time corresponding to the field with the same name but different values. Then, a unified data view is generated based on the entity alignment results and the effective range of the attribute, so that the unified data view retains the attribute content of the same entity in different time ranges.

[0049] The effective range of attributes is determined based on the record times of fields with the same name but different values ​​within the entity alignment results, including: Read field records belonging to the same entity record group in the entity alignment result, and treat field records with the same field name but different field values ​​as field records with the same name but different values; Based on the business records corresponding to the same name but different values ​​field records, read the corresponding entity association path, and write the records of the same name but different values ​​field records into the corresponding entity association path at any time. Determine the effective range of attributes for records with the same name but different values ​​based on their position in the corresponding entity's association path at the time of the record.

[0050] In this embodiment, the field record includes the field name, field value, record time, and the business record to which the field record belongs. The same name but different values ​​field record refers to the field record with the same field name but different field value within the same entity record group. The corresponding entity association path refers to the entity association path formed during the entity alignment process of the business record to which the same name but different values ​​field record belongs.

[0051] Furthermore, after reading the field records belonging to the same entity record group in the entity alignment result, the field records with the same field name but different field values ​​are regarded as same-named but different-value field records. Then, according to the business records corresponding to the same-named but different-value field records, the corresponding entity association path is read, and the recording time of the same-named but different-value field records is written into the corresponding entity association path. The time range of the field value is limited by the position of the recording time in the corresponding entity association path.

[0052] Records with the same name but different values ​​in a field will be written to the corresponding entity's association path at the specified time, including: Read the record time of the corresponding business record in the same name but different value field, and read the record position of the corresponding business record in the entity association path in the same name but different value field. Read adjacent business records based on their record positions, and match the record times of adjacent business records with the record times of the corresponding business records with the same name but different values. Record the record time of the corresponding business record for the same name but different values ​​field and write it to the record location to obtain the corresponding entity association path at the record time.

[0053] In this embodiment, the record position refers to the arrangement position of the business record corresponding to the same name but different value field record in the entity association path. Adjacent business records refer to business records located before or after the record position in the entity association path. Correspondence means that the record time of adjacent business records is matched with the record time of the business record corresponding to the same name but different value field record according to the arrangement direction in the entity association path.

[0054] It should be noted that by reading the record position of the corresponding business record of the same name but different value field record, we can avoid directly dividing the attribute effective range based solely on the size of the record time. Instead, the record time is placed in the business connection relationship corresponding to the entity association path for processing. When the record time of the corresponding business record of the same name but different value field record is marked and written to the record position, the corresponding entity association path simultaneously contains the business connection position and the record time position.

[0055] A unified data view is generated based on the entity alignment results and the attribute effective range, including: Read the entity record group from the entity alignment result and group the multi-source business records in the entity record group by field name; Read the attribute validity range and write the field name, field value, and attribute validity range into the corresponding entity record group; A unified data view is generated based on the entity record group whose field names, field values, and attribute validity ranges are written.

[0056] In this embodiment, grouping by field name means organizing field records from different data sources within the same entity record group into the same field item according to the field name. The corresponding entity record group refers to the entity record group to which the field name, field value, and attribute effective range belong.

[0057] Understandably, when generating a unified data view, the entity record group in the entity alignment result is read first, then the multi-source business records in the entity record group are grouped by field name, then the attribute effective range is read, and the field name, field value and attribute effective range are written into the corresponding entity record group. Finally, the unified data view is generated by the entity record group with the field name, field value and attribute effective range written into it, so that the multi-source field content of the same entity is organized according to the field name and effective time range.

[0058] Example 2

[0059] Please see Figure 2 As shown, based on the same inventive concept, this embodiment discloses a data integration and utilization system based on multiple data sources. For details not covered in this embodiment, please refer to the relevant parts of Embodiment 1. The system includes: Entity grouping module: used to obtain multi-source business records, deterministic identifiers, record times and entity association records, and group multi-source business records with the same deterministic identifier into the same entity record group; Field mapping module: used to determine cross-source field mapping relationships based on the field positions of fields with the same value and adjacent field positions within an entity record group; Relationship extension module: used to obtain entity association paths by connecting multi-source business records outside the entity association record group with the entity record group, and to determine candidate entity groups by combining entity association paths and cross-source field mapping relationships; View generation module: It is used to merge candidate entity groups into entity record groups, update cross-source field mapping relationships according to the merged entity record groups, until entity record groups no longer increase, obtain entity alignment results, determine the attribute effective range according to the record time of fields with the same name but different values ​​in the entity alignment results, and generate a unified data view based on the entity alignment results and attribute effective ranges.

[0060] The detailed description above, in conjunction with the accompanying drawings, describes examples but does not represent all examples that can be implemented or fall within the scope of the claims. The terms “example” and “exemplary” are used in this specification to mean “serving as an example, instance or illustration” and do not mean “superior to or better than other examples”.

[0061] Throughout this specification, the phrase "an embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. Therefore, the use of these phrases may refer to more than one embodiment. Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0062] It should also be noted that these embodiments may be described as processes depicted as flowcharts, structural diagrams, or block diagrams. Although a flowchart may describe the operations as sequential processes, many of these operations can be performed in parallel or concurrently, and the order of these operations may be rearranged.

Claims

1. A method for data integration and utilization based on multiple data sources, characterized in that, include: Obtain multi-source business records, deterministic identifiers, record times, and entity association records, and group multi-source business records with the same deterministic identifier into the same entity record group; Determine the cross-source field mapping relationship based on the field position of fields with the same value and the position of adjacent fields within the entity record group; Based on the entity association record connection group, the entity association path is obtained by connecting the multi-source business records outside the entity association record group with the entity record group. The candidate entity group is determined by combining the entity association path and the cross-source field mapping relationship. The candidate entity group is merged into the entity record group. The cross-source field mapping relationship is updated according to the merged entity record group until the entity record group no longer increases, and the entity alignment result is obtained. The attribute effective range is determined according to the record time of the same name but different value fields in the entity alignment result. A unified data view is generated based on the entity alignment result and the attribute effective range.

2. The data integration and utilization method based on multiple data sources according to claim 1, characterized in that, Based on the field positions of fields with the same value and adjacent fields within the entity record group, determine the cross-source field mapping relationship, including: Read multi-source business records belonging to different data sources within an entity record group, and extract field values ​​based on field position; retain field records that have both field position and field value. Pair records with the same field value but different data sources; write the paired records into pairs of fields with the same value. Read the positions of the adjacent fields on both sides of the same value field pair to obtain the adjacent field position group; Write the field position of the same value field pair and the adjacent field position group to the field position record; associate the field position record with the corresponding entity record group; Determine cross-source field mapping relationships based on the field position records that appear repeatedly in different entity record groups.

3. The data integration and utilization method based on multiple data sources according to claim 2, characterized in that, Read the positions of the adjacent fields on both sides of the same value field pair to obtain the adjacent field position group, including: Based on the field positions of the same value field pairs in different data sources, read the first and second field positions respectively; write the read source to the first and second field positions; Read the field type corresponding to the preceding field position and the field type corresponding to the following field position; associate the field type with the corresponding preceding and following field positions; By combining the positions of the preceding and following fields that are of the same type and have the same relative direction, we obtain adjacent field position groups.

4. The data integration and utilization method based on multiple data sources according to claim 1, characterized in that, Based on the entity association record connection group, the entity association path is obtained by linking the multi-source business records outside the entity association record group with the entity record group, including: Read multi-source business records that are not assigned to an entity record group to obtain the starting record; read the associated fields and record time in the starting record; Based on the associated fields in the starting record, retrieve the referenced record from the entity's associated records; then connect the referenced record and the starting record. Connect the starting record and the entity record group in the entity association record to obtain the forward record segment and the reverse record segment; Records with the same business records in the forward and reverse record segments but opposite connection directions are taken as path intersection records, and the path intersection records are associated with the starting record. The entity association path is obtained by connecting the starting record, the path intersection record, and the entity record group in the order of connection.

5. The data integration and utilization method based on multiple data sources according to claim 4, characterized in that, By concatenating the starting record and the entity record group in the entity association record, we obtain the forward record segment and the reverse record segment, including: Starting from the associated field of the starting record, read the referenced record with the same field value from the entity's associated record; write the connection relationship between the starting record and the referenced record into the forward path record; Starting from the multi-source business records within the entity record group, read the source records pointing to multi-source business records outside the group from the entity association records, and write the connection relationship between the multi-source business records and the source records within the entity record group into the reverse path record; Read the recording times from the forward path records and the reverse path records respectively; arrange the forward path records and the reverse path records according to the recording times; Based on the arranged forward path records and reverse path records, the forward record segment and the reverse record segment are obtained.

6. The data integration and utilization method based on multiple data sources according to claim 5, characterized in that, Write the connection relationship between the starting record and the record it points to into the forward path record, including: Read the associated fields in the starting record to obtain the value of the starting associated field, and search for the record that is pointed to in the entity associated records with the same field value as the value of the starting associated field; Read the record time and field position of the pointed-to record, and write the record time and field position of the pointed-to record into the pointed-to record item; Write the starting record, the record being pointed to, and the connection direction between the starting record and the record being pointed to into the same record to obtain the forward path record.

7. The data integration and utilization method based on multiple data sources according to claim 1, characterized in that, The effective range of attributes is determined based on the record times of fields with the same name but different values ​​within the entity alignment results, including: Read field records belonging to the same entity record group in the entity alignment result, and treat field records with the same field name but different field values ​​as field records with the same name but different values; Based on the business records corresponding to the same name but different values ​​field records, read the corresponding entity association path, and write the records of the same name but different values ​​field records into the corresponding entity association path at any time. Determine the effective range of attributes for records with the same name but different values ​​based on their position in the corresponding entity's association path at the time of the record.

8. The data integration and utilization method based on multiple data sources according to claim 7, characterized in that, Records with the same name but different values ​​in a field will be written to the corresponding entity's association path at the specified time, including: Read the record time of the corresponding business record in the same name but different value field, and read the record position of the corresponding business record in the entity association path in the same name but different value field. Read adjacent business records based on their record positions, and match the record times of adjacent business records with the record times of the corresponding business records with the same name but different values. Record the record time of the corresponding business record for the same-named but different-valued field and write it to the record location to obtain the corresponding entity association path at the record time.

9. A data integration and utilization method based on multiple data sources according to claim 1, characterized in that, A unified data view is generated based on the entity alignment results and the attribute effective range, including: Read the entity record group from the entity alignment result and group the multi-source business records in the entity record group by field name; Read the attribute validity range and write the field name, field value, and attribute validity range into the corresponding entity record group; A unified data view is generated based on the entity record group whose field names, field values, and attribute validity ranges are written.

Images