Heterogeneous record fusion method and device based on double timestamps, equipment and medium

Abstract

The application discloses a heterogeneous record fusion method and device based on double timestamps, equipment and medium, relates to the technical field of database data processing, and the method comprises the following steps: acquiring multi-source heterogeneous records composed of source identification, object identification, event type, homologous double timestamps, version information and content; generating a time anchor pair according to a reference source and an anchor mapping table, determining a clock offset value from a cross-source event timestamp difference; determining a write delay value from a homologous double timestamp difference, and generating a corrected time interval in combination with the clock offset value; associating cross-source records to form an event record group according to the object identification, the event type, the content and the corrected time interval; constructing a bearing relationship according to the version information, determining current effective records and conflict fields; fusing non-conflict fields, establishing branches according to conflict values, and obtaining fused event records. The application can associate and fuse cross-source records corresponding to the same event under the conditions of clock offset, write delay and version evolution of multi-source heterogeneous records.

Description

Technical Field

[0001] This application relates to the field of database data processing technology, and in particular to a method, apparatus, device and medium for heterogeneous record fusion based on dual timestamps. Background Technology

[0002] As data exchange between different data systems increases, the same business event may generate records with different field structures, time information, and version information in multiple data sources. It is necessary to associate and merge these heterogeneous records from multiple sources to support unified storage and subsequent data processing.

[0003] Current practices typically involve first mapping fields from different data sources, then associating records based on business object identifiers, event types, and timestamp differences, and finally selecting records to be merged according to their generation time or version order, merging fields with the same name.

[0004] However, clock skew may exist between different data sources, and there is also a write latency between the event timestamp and the write timestamp. After the same record is modified, undone, or branched, multiple successor versions may also be formed, making it difficult to determine the same event and its current version based solely on the proximity of timestamps or the last write record. Therefore, how to associate and merge cross-source records corresponding to the same event in the context of clock skew, write latency, and version evolution among multi-source heterogeneous records has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, device and medium for heterogeneous record fusion based on dual timestamps, which aims to solve the technical problem of how to associate and fuse cross-source records corresponding to the same event when there are clock offsets, write delays and version evolutions in multi-source heterogeneous records.

[0006] To achieve the above objectives, this application proposes a heterogeneous record fusion method based on dual timestamps, the method comprising: Acquire multi-source heterogeneous record data, which includes data source identifier, business object identifier, business event type, dual timestamp, record version information, and record content. The dual timestamp consists of an event timestamp and a write timestamp generated by the same data source clock. Time anchor pairs are generated based on the preset reference data source and anchor mapping table, and the clock offset is determined based on the difference in timestamps of cross-source events. The write latency is determined based on the difference between the two timestamps from the same source, and a correction time interval is generated based on the clock offset and the write latency. Based on the business object identifier, the business event type, the record content, and the correction time interval, the heterogeneous record data is cross-sourced to obtain an event record group; Based on the record version information, construct the version succession relationship within the event record group, and determine the currently valid records and conflict fields; By merging the record fields other than the conflicting field in the existing valid records, a conflict value branch of the conflicting field is established to obtain the merged event record.

[0007] Furthermore, to achieve the above objectives, this application also proposes a heterogeneous record fusion device based on dual timestamps, the device comprising: The data acquisition module is used to acquire multi-source heterogeneous record data. The heterogeneous record data includes data source identifier, business object identifier, business event type, dual timestamp, record version information and record content. The dual timestamp consists of an event timestamp and a write timestamp generated by the same data source clock. The anchor pairing module is used to generate time anchor pairs based on a preset reference data source and anchor mapping table, and to determine the clock offset based on the difference in timestamps of cross-source events. The time correction module is used to determine the write latency based on the difference between the two timestamps from the same source, and to generate a correction time interval based on the clock offset and the write latency. The event association module is used to perform cross-source association of the heterogeneous record data based on the business object identifier, the business event type, the record content, and the correction time interval to obtain an event record group; The version processing module is used to construct the version succession relationship within the event record group based on the record version information, and to determine the currently valid records and conflict fields; The record fusion module is used to fuse record fields other than the conflicting field in the existing valid records, establish conflict value branches of the conflicting field, and obtain fused event records.

[0008] Furthermore, to achieve the above objectives, this application also proposes a heterogeneous record fusion device based on dual timestamps, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the heterogeneous record fusion method based on dual timestamps as described above.

[0009] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the heterogeneous record fusion method based on dual timestamps as described above.

[0010] One or more technical solutions proposed in this application have at least the following technical effects: This process acquires heterogeneous record data from multiple sources, representing records from different data sources as a data structure containing a data source identifier, business object identifier, business event type, dual timestamps, record version information, and record content. The event timestamp and write timestamp are generated by the same data source clock, providing a unified data foundation for subsequent time correction, event association, and version processing. Based on a preset reference data source and anchor mapping table, records corresponding to the same event time point from different data sources are paired as time anchor points. The clock offset is determined using the cross-source event timestamp difference, reducing the impact of clock differences between data sources on record time comparisons. The write latency is determined based on the difference between the event timestamp and write timestamp within the same data source, and then combined with the time... Clock offset generation and correction time intervals enable the recorded time to simultaneously reflect clock offset and write latency. Cross-source association is performed based on business object identifier, business event type, record content, and correction time interval, grouping records corresponding to the same event into event record groups, reducing the possibility of erroneous associations caused by relying solely on identifiers or single timestamps. Version succession relationships within event record groups are constructed based on record version information, identifying currently valid records and conflicting fields from different versions, preventing records already inherited by subsequent versions from directly participating in fusion. Non-conflicting fields in currently valid records are merged, and conflicting value branches with different field values ​​are retained for conflicting fields, resulting in fused event records, avoiding direct overwriting of record content when field values ​​are inconsistent. Therefore, this application can associate and merge cross-source records corresponding to the same event even when there are clock offsets, write latency, and version evolutions in multi-source heterogeneous records, forming a unified event record that retains valid fields, version relationships, and conflicting field values, facilitating subsequent querying, verification, and processing of cross-system event data. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the first embodiment of the heterogeneous record fusion method based on dual timestamps in this application. Figure 2 This is a flowchart illustrating the second embodiment of the heterogeneous record fusion method based on dual timestamps in this application. Figure 3This is a schematic diagram of the module structure of the heterogeneous record fusion device based on dual timestamps according to an embodiment of this application; Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the heterogeneous record fusion method based on dual timestamps in the embodiments of this application.

[0014] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0016] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0017] It should be noted that the executing entity in this application embodiment can be a server, workstation, database node, or heterogeneous record fusion system with data processing, data storage, and network communication functions. The following description uses a heterogeneous record fusion system as an example.

[0018] Based on this, the first embodiment of this application provides a heterogeneous record fusion method based on dual timestamps, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the heterogeneous record fusion method based on dual timestamps of this application.

[0019] In this embodiment, the heterogeneous record fusion method based on dual timestamps includes steps S10 to S60: Step S10: Obtain multi-source heterogeneous record data. The heterogeneous record data includes data source identifier, business object identifier, business event type, dual timestamp, record version information, and record content. The dual timestamp consists of an event timestamp and a write timestamp generated by the same data source clock. It should be noted that multi-source heterogeneous record data refers to data from multiple data sources with differences in record structure, time field, or version representation; data source identifier refers to the identifier used to distinguish the source of the record; business object identifier refers to the identifier used to represent the business object corresponding to the record; business event type refers to the information used to represent the event category corresponding to the record; dual timestamp refers to the event timestamp generated by the clock of the same data source and the write timestamp; record version information refers to the data that characterizes the record version and version changes; record content refers to the fields and field values ​​in the record that carry business data.

[0020] Understandably, the heterogeneous record fusion system receives records from different data sources. According to the data access configuration of each heterogeneous data system, it reads the data source identifier, business object identifier, business event type, time field, record version information, and record content. It checks the clock source corresponding to the event timestamp and the write timestamp, combining the event timestamp and write timestamp generated by the same data source clock into a dual timestamp. Then, it encapsulates each field into multi-source heterogeneous record data according to the record correspondence. This step preserves the source information, event time information, write time information, and version information of different data sources, providing a data foundation for subsequent time correction, event association, and version processing.

[0021] Step S20: Generate time anchor pairs based on the preset reference data source and anchor mapping table, and determine the clock offset based on the cross-source event timestamp difference; It should be noted that the preset reference data source refers to the data source used as the benchmark for cross-source time comparison; the anchor mapping table refers to the configuration data that records the anchor events and corresponding field relationships between different data sources; the time anchor pair refers to two records in different data sources that correspond to the same event time point; the cross-source event timestamp difference refers to the difference between the timestamps of the two events in the time anchor pair; and the clock offset refers to the time deviation of the non-reference data source clock relative to the reference data source clock.

[0022] Understandably, the process involves determining a time comparison benchmark based on a preset reference data source, searching for records corresponding to the same event time point between the reference data source records and other data source records according to the anchor point mapping table, and forming time anchor point pairs. The cross-source event timestamp differences within the time anchor point pairs are calculated, and then these differences are aggregated according to data source to obtain the clock offset of each data source relative to the reference data source. This step utilizes time records of the same event time point in different data sources to determine the clock offset, enabling the time information from different data sources to be converted to a unified time benchmark.

[0023] Step S30: Determine the write latency based on the difference between the two timestamps from the same source, and generate a correction time interval based on the clock offset and the write latency; It should be noted that the difference between the same source timestamp and the event timestamp in the same record refers to the difference between the write timestamp and the event timestamp; the write latency refers to the time interval between the formation of the event and the writing of the corresponding record to the data source; and the corrected time interval refers to the range of event times formed after time conversion of the event timestamp and the write timestamp of the record.

[0024] Understandably, the process involves calculating the difference between the two timestamps for each record within the same data source, summarizing these differences according to data source and event category to obtain the corresponding write latency; converting the event timestamp using the clock offset, and then converting the write timestamp using the clock offset and write latency; finally, forming the corrected time interval for each record based on the conversion results of the two timestamps. This step uses the clock offset between data sources and the write latency within the data source for time conversion, ensuring that the corrected time interval reflects the time range of the corresponding event for each record.

[0025] Step S40: Perform cross-source association on the heterogeneous record data according to the business object identifier, the business event type, the record content, and the correction time interval to obtain an event record group; It should be noted that cross-source association refers to the process of establishing a correspondence between records from different data sources for the same event; an event record group refers to a set of records that are identified as corresponding to the same event through cross-source association.

[0026] Understandably, the process involves filtering cross-source records from multi-source heterogeneous record data based on business object identifiers and business event types, identifying records that might correspond to the same event, and then comparing the record content and corrected time intervals of these cross-source records. When the record content meets preset association conditions and the corrected time interval meets preset time association conditions, a cross-source association is established between the corresponding records, grouping records corresponding to the same event into the same event record group. This step utilizes business objects, event categories, record content, and corrected time ranges to establish cross-source associations, reducing erroneous associations caused by relying solely on a single timestamp to associate records.

[0027] Step S50: Construct the version succession relationship within the event record group based on the record version information, and determine the currently valid records and conflict fields; It should be noted that version succession refers to the sequential relationship between different version records within an event record group. Currently valid records refer to those currently participating in record merging, determined according to version succession and preset version status processing rules. This includes records in their current state, as well as historical state records selected by tracing back along the preceding version identifier when the final version processing fails. Conflicting fields refer to fields in different currently valid records that have the same business meaning but inconsistent values.

[0028] Understandably, the process involves reading the record version information of each record within the event log group, connecting the corresponding records based on the version sequence indicated by the version information to form a version succession relationship, identifying the currently valid records participating in the fusion along the version succession relationship, and then comparing the currently valid records corresponding to different data sources. Records with the same field meaning but inconsistent field values ​​are identified as conflicting fields. This step handles record version changes before field fusion, reducing the possibility of historical or non-current versions participating in the fusion process, and distinguishing between record fields that can be directly fused and conflicting fields whose different field values ​​need to be retained.

[0029] Step S60: Merge the record fields in the existing valid records except for the conflict field, establish the conflict value branch of the conflict field, and obtain the merged event record.

[0030] It should be noted that non-conflicting fields refer to record fields in the current valid records that have not been identified as conflicting fields; conflicting value branches refer to storage branches established for different field values ​​of the same conflicting field; and merged event records refer to records formed after merging fields of the current valid records corresponding to the same event.

[0031] Understandably, the process involves extracting all record fields from currently valid records, excluding conflicting fields, and merging and storing identical field values ​​corresponding to the same field. For each conflicting field, conflict value branches are created based on different field values, and each field value is retained in its corresponding branch. The business object identifier, business event type, merged non-conflicting fields, and conflict value branches are then associated to form a merged event record. This step merges record fields that can be consistently expressed, while retaining the different field values ​​of conflicting fields in a branching manner, reducing information loss caused by a single field value overwriting other field values ​​during the record merging process.

[0032] This embodiment determines the clock offset between different data sources by using time anchor points, determines the write latency by using the difference between the two timestamps from the same source, and then establishes cross-source event associations using the correction time interval, business object identifier, business event type, and record content. Within the event record group, the currently valid records are determined according to the version succession relationship, non-conflicting fields are merged, and different field values ​​are retained by the conflict value branch. This improves the determineability of cross-source event correspondence and the traceability of the source of merged records, and reduces the impact of clock offset, write latency, historical version, and field value overwriting on the record fusion results.

[0033] As an example, the steps for acquiring multi-source heterogeneous record data include: receiving source records from various heterogeneous data systems and configuring data source identifiers; converting the source object identifiers and source event types in the source records into business object identifiers and business event types respectively, according to a preset object identifier mapping table and a preset event type mapping table; extracting the event timestamp and write timestamp generated by the same data source clock from the source records to obtain dual timestamps; extracting the version family identifier, version identifier, previous version identifier, and version status from the source records to obtain record version information, and encapsulating the business fields and field values ​​into record content; associating the data source identifier, the business object identifier, the business event type, the dual timestamps, the record version information, and the record content to obtain heterogeneous record data.

[0034] It should be noted that heterogeneous data systems can be data systems with different data structures, field naming methods, or version recording methods, such as business master systems, process recording systems, detection result systems, and data archiving systems. A source record refers to the original record received from the heterogeneous data system that has not yet undergone field conversion; a preset object identifier mapping table is a configuration table that records the correspondence between data source identifiers, source object identifiers, and business object identifiers; a preset event type mapping table is a configuration table that records the correspondence between data source identifiers, source event types, and business event types; a clock domain identifier is an identifier used to represent the system clock service corresponding to the time field; and a version family identifier is an identifier used to aggregate different versions of the same record.

[0035] A version identifier is a unique identifier used to distinguish record versions within the same data source. When the source record uses a version sequence number that is unique only within a version family, the version family identifier is associated with the version sequence number to form the version identifier. The predecessor version identifier refers to the version identifier corresponding to the previous version directly inherited by the current version. Version status refers to information indicating the record version processing result, which can be current status, historical status, deleted status, revoked status, verification failure status, or generation failure status. Current status indicates that the record is currently valid; historical status indicates that the record has been inherited by a subsequent version; deleted status and revoked status indicate that the record has been deleted or withdrawn, respectively; verification failure status and generation failure status indicate that the record failed verification or was not generated, respectively.

[0036] Understandably, firstly, a data source identifier is registered for each heterogeneous data system in the data access configuration, and the correspondence between the data access channel and the data source identifier is saved; when receiving a source record, the corresponding data source identifier is queried according to the actual access channel and written into the source record. The same data source identifier is used when the same heterogeneous data system sends records through multiple data access channels, while different heterogeneous data systems use different data source identifiers.

[0037] The preset object identifier mapping table includes data source identifier, source object identifier, business object identifier, and mapping status. The preset event type mapping table includes data source identifier, source event type, business event type, and mapping status. The preset object identifier mapping table is queried based on the data source identifier and source object identifier, and the retrieved business object identifier is written to the source record. Similarly, the preset event type mapping table is queried based on the data source identifier and source event type, and the retrieved business event type is written to the source record. If no valid mapping relationship is found, or if multiple valid mapping relationships are found, the corresponding source record is marked as having an identifier mapping error, and the corresponding heterogeneous record data is not output. The mapping status indicates whether the corresponding mapping relationship participates in the current processing; only entries with an enabled mapping status are considered valid mapping relationships.

[0038] Then, in the data access configuration of each heterogeneous data system, the event timestamp field, write timestamp field, clock domain identifier, time zone, time unit, and time precision are recorded. The clock domain identifiers of the two timestamp fields are read. If the clock domain identifiers are the same, the event timestamp and write timestamp are converted into a unified time unit and a unified time representation according to the data access configuration. Then, the converted event timestamp and write timestamp are combined into a double timestamp. When heterogeneous data systems use the same system clock service to generate two timestamps, the same clock domain identifier can be configured for the two timestamp fields in the data access configuration. If any timestamp field is missing, the time value format is incorrect, or the two clock domain identifiers are different, the corresponding source record is marked as having a time field error, and the corresponding heterogeneous record data is not output.

[0039] Next, according to the position of the version field in the data access configuration, the version family identifier, version identifier, predecessor version identifier, and version status are read from the source record and encapsulated into record version information according to the field correspondence of the same source record. The predecessor version identifier of the initial version is allowed to be empty. If the predecessor version identifier is missing for non-initial versions, or if the source record is missing at least one of the version family identifier, version identifier, and version status, the corresponding source record is marked as having an abnormal version field. The data source field, object identifier field, event type field, time field, and version field are excluded from the source record from which the field extraction has been completed, and the remaining business fields and field values ​​are encapsulated into record content.

[0040] Finally, the data source identifier, business object identifier, business event type, dual timestamp, record version information and record content corresponding to the same source record are written into the same record object to form heterogeneous record data; for source records with abnormal identifier mapping, abnormal time field or abnormal version field, the abnormality type and missing field information are saved and stored separately from the heterogeneous record data that has been formed.

[0041] This example configures the record source through the data access channel, unifies cross-source identification through object identifier mapping table and event type mapping table, and checks the clock source and record version field of dual timestamps. This reduces the number of records with source confusion, inconsistent object identifiers, incomparable time fields, or missing version information entering the subsequent processing process, while forming heterogeneous record data with relatively uniform field structure.

[0042] As an example, the step of determining the write latency based on the difference between the two timestamps from the same source, and generating a correction time interval based on the clock offset and the write latency, includes: for each record, subtracting the corresponding clock offset from the event timestamp to obtain a first correction time; dividing the heterogeneous record data according to the data source identifier, business event type, and a preset statistical time window to which the first correction time belongs; calculating the difference between the write timestamp and the event timestamp in each division result, and determining the median of the obtained difference as the write latency; subtracting the corresponding clock offset and the write latency from the write timestamp of each record to obtain a second correction time; subtracting a preset time tolerance from the earlier time between the first correction time and the second correction time, and adding the preset time tolerance to the later time to obtain a correction time interval.

[0043] It should be noted that the first correction time refers to the time obtained after converting the event timestamp to the reference data source time base using the clock offset; the second correction time refers to the time obtained after converting the write timestamp to the reference data source time base using the clock offset and write latency; the preset statistical time window refers to a window that divides records or time anchor pairs according to a time range; in this example, the preset statistical time window is divided into a clock offset statistical time window and a write latency statistical time window according to its purpose. The former divides time anchor pairs based on the event timestamps of records from non-reference data sources, while the latter divides heterogeneous record data based on the first correction time of the records. The preset time tolerance refers to the interval expansion amount set according to the historical deviation between two correction times.

[0044] It is understandable that the set is based on the time anchor points of data source s and reference data source r within the p-th statistical time window. For example, the formula for calculating clock offset is: in, This represents the clock offset of data source s within the p-th statistical time window, where p represents the sequence number of the clock offset statistical time window. and These represent the event timestamps of the non-reference data source record and the reference data source record in the i-th time anchor point pair, respectively, with the clock offset corresponding to the reference data source set to 0. Each clock offset is associated with and stored with the data source identifier and the corresponding statistical time window for use by records within the same time range of the same data source.

[0045] For the j-th record in data source s, read the clock offset according to the statistical time window corresponding to the data source identifier and the event timestamp, and subtract the clock offset from the event timestamp to obtain the first correction time: in, This represents the event timestamp of the j-th record. p(j) represents the clock offset statistics time window to which the event timestamp of the j-th record belongs; the length of the clock offset statistics time window is set according to the historical clock offset change frequency of the data source and the generation frequency of time anchor pairs, and can be from 30 min to 24 h, and is 1 h in this example.

[0046] Then, the heterogeneous record data is segmented according to the data source identifier, business event type, and the statistical time window in which the first correction time occurs, to determine the segmentation results. This represents the set of records corresponding to the data source s, business event type c, and the q-th write latency statistics time window. When the write timestamp of a record is earlier than the event timestamp, the corresponding record is marked as having a time sequence error and is not included in the write latency statistics process.

[0047] The formula for calculating write latency is: in, This represents the write latency of the corresponding partition result, where q represents the sequence number of the write latency statistics time window. and These represent the event timestamp and write timestamp of the h-th record, respectively, where h represents the record number in the write delay statistics record set; the superscript ev represents the event timestamp, and the superscript wr represents the write timestamp. The preset minimum number of records in the statistics window can be between 5 and 30, and is set to 10 in this example. When the number of records is less than the preset minimum number of records, the current statistics time window is merged with statistics time windows that are adjacent in time and have the same data source identifier and business event type. If the preset minimum number of records is still not met after merging, the nearest prior valid write delay is called. If the prior valid write delay does not exist, the corresponding record is marked as having insufficient delayed data.

[0048] Based on this, the clock offset and write delay are subtracted from the write timestamp of the j-th record to obtain the second correction time: in, This represents the write timestamp of the j-th record. Let q(j) represent the second correction time of the j-th record, and q(j) represent the write latency statistics time window to which the first correction time of the j-th record belongs. The earlier of the first and second correction times is taken as the starting point of the interval reference, and the later time is taken as the ending point of the interval reference, and a preset time tolerance is extended outwards accordingly. The correction time interval is obtained as follows: in, This represents the correction time interval for the j-th record. This indicates the preset time tolerance corresponding to business event type c in data source s.

[0049] Preset time tolerance It can retrieve the historical records corresponding to business event type c in data source s. The 90th to 99th percentile is used, and in this example, the 95th percentile is used. When the number of historical records is insufficient, the preset time tolerance is the sum of the data source timestamp resolution and the upper limit of the clock synchronization error.

[0050] For records for which the corresponding clock offset or write latency is not obtained, no correction time interval is generated, and the corresponding record is stored separately as a record with insufficient time parameters, and the corresponding record is not included in the cross-source association of the current batch; after obtaining the corresponding clock offset and write latency, the corresponding record with insufficient time parameters is reprocessed.

[0051] This example converts event timestamps from different data sources to a reference time base using clock offset, performs time backtracking on the write timestamp using write latency, and forms a correction time interval using two correction moments and a preset time tolerance. This allows the recording time to simultaneously reflect cross-source clock offset, data write latency, and latency fluctuations, reducing the impact of a single timestamp deviation on subsequent cross-source record association.

[0052] As an example, the step of performing cross-source association on the heterogeneous record data based on the business object identifier, the business event type, the record content, and the correction time interval to obtain an event record group includes: distinguishing reference data source records and non-reference data source records according to a preset reference data source, and forming candidate record pairs with reference data source records and non-reference data source records that have the same business object identifier and business event type and intersecting correction time intervals; extracting common comparison fields of the candidate record pairs according to a preset comparison field table, and unifying the field value format; when the common comparison field is not empty, determining the content matching degree as the ratio of the number of common comparison fields with the same field value to the total number of common comparison fields; for each non-reference data source record, establishing an event association between records within the candidate record pair when the candidate record pair with the highest content matching degree is unique and the maximum content matching degree is not less than a preset matching degree threshold; and forming an event record group with the same reference data source record and its corresponding associated non-reference data source record.

[0053] It should be noted that: a reference data source record refers to a record whose data source identifier corresponds to a preset reference data source; a non-reference data source record refers to a record whose data source identifier does not correspond to a preset reference data source; a candidate record pair refers to two records with the same business object identifier and business event type, overlapping correction time intervals, and originating from a reference data source and a non-reference data source respectively; a preset comparison field table refers to a configuration table that records business event type, reference field name, non-reference field name, field data type, and uniform format; common comparison fields refer to fields in the candidate record pair that both have non-empty field values ​​and have a corresponding relationship; content matching degree refers to the ratio of the number of common comparison fields with the same field value to the total number of common comparison fields; and a preset matching degree threshold refers to the minimum content matching degree required to establish an event association.

[0054] Understandably, firstly, reference data source records and non-reference data source records are distinguished based on the data source identifier; for each non-reference data source record, reference data source records with the same business object identifier and business event type are selected, and then it is determined whether the correction time intervals of the two records intersect; the correction time interval of reference data source record a is used as the basis for the next step. Correction time interval for non-reference data source record b For example, two records will be paired as candidate record pairs if the following condition is met: in, and These represent the start and end times of the correction time interval for reference data source record a, respectively. and These represent the start and end times of the correction time interval for record b from the non-reference data source, respectively.

[0055] The preset comparison field table sets the field correspondence according to the business event type. The same table item includes the reference field name, non-reference field name, field data type, and uniform format. For each candidate record pair, the corresponding table item is read according to the business event type, and the corresponding field value is extracted from the record content of the two records. When both records have non-empty field values, the corresponding field is used as the common comparison field. When only one record has a non-empty field value or neither record has a field value, the corresponding field is not included in the common comparison field.

[0056] Then, the field value format of the common comparison field is unified according to the preset comparison field table; the coding field is converted into a unified code according to the coding correspondence in the table, the numerical field is converted into a unified unit of measurement and comparison precision, the time field is converted into a unified time zone and time precision, and the text field is unified in character encoding and the leading and trailing whitespace characters of the field value are removed; after the field value format is unified, the two field values ​​of the same common comparison field are compared according to the field data type.

[0057] When candidate record pairs have common comparison fields, count the number of common comparison fields with the same field value and calculate the content matching degree: in, This indicates the degree of content matching between the reference data source record a and the non-reference data source record b. This indicates the number of fields that are compared together with the same field value. This indicates the total number of fields compared together.

[0058] For each non-reference data source record, compare the content matching degree of the corresponding candidate record pair. When the maximum content matching degree corresponds to a unique candidate record pair and is not less than the preset matching degree threshold, establish an event association between the two records in the candidate record pair. When there is no candidate record pair, the candidate record pair does not have a common comparison field, the maximum content matching degree is less than the preset matching degree threshold, or multiple candidate record pairs have the same maximum content matching degree, do not establish an event association for the corresponding non-reference data source record. The preset matching degree threshold can be set from 0.7 to 0.95. In this example, it is set to 0.8. The specific value is set according to the distribution of content matching degree in the history of confirmed event correspondence.

[0059] Finally, for each reference data source record with an established event association, all non-reference data source records with event associations with the reference data source record are queried, and the reference data source record and the queried non-reference data source records are written into the same event record group. In one implementation, before constructing the version inheritance relationship, based on the data source identifier and version family identifier of the associated records, other records with the same data source identifier, version family identifier, business object identifier, and business event type are queried from the heterogeneous record data of the generated correction time interval, and the queried records are added to the corresponding event record group; records without established event associations and that do not meet the version record addition conditions are marked as unassociated records and are not included in the event record group.

[0060] This example filters cross-source records by business object identifier, business event type, and correction time interval, and then uses the record content to determine the content matching degree of candidate record pairs. Only records with unique matching results and meeting the preset matching degree threshold are associated with events, which reduces the false association of records caused by multiple similar events within a similar time period, and makes the event record group include the reference data source record as the association benchmark and the corresponding non-reference data source record.

[0061] As an example, the steps of constructing the version succession relationship within the event record group based on the record version information and determining the currently valid records and conflicting fields include: extracting the version family identifier, version identifier, predecessor version identifier, and version status from the record version information, and dividing the records within the event record group according to the data source identifier and the version family identifier to obtain the source record version group; using the version identifier of the records in the source record version group as the version node, and connecting the version nodes according to the predecessor version identifier to obtain the version succession relationship; determining the version node that is not referenced by other version nodes as a predecessor version as the end version node, and starting from the end version node, querying along the predecessor version identifier according to the preset version status processing rules to determine the currently valid records; aligning the record fields in the currently valid records according to the preset field conversion table, and converting the field values ​​to a preset unified format; when the non-empty field values ​​of the same-named record fields in at least two of the currently valid records are different, the same-named record fields are determined to be conflicting fields.

[0062] It should be noted that: a source record version group refers to a set of records within an event record group that share the same data source identifier and version family identifier; a version node refers to a node that uses a version identifier as its node marker and is associated with a corresponding record; a terminal version node refers to a version node that is not referenced by other version nodes as a preceding version; a version branch refers to a version path that connects the starting version node to the terminal version node; preset version state processing rules refer to the configuration rules that record the selection, stopping, or backtracking processing methods corresponding to different version states; and preset field conversion table refers to the configuration table that records the correspondence between data source identifier, source field name, unified field name, field data type, and unified format.

[0063] Understandably, firstly, the version family identifier, version identifier, previous version identifier, and version status are read from the record version information of each record in the event record group, and the records are divided according to the data source identifier and the version family identifier; records with the same data source identifier and version family identifier are placed into the same source record version group, and records corresponding to different data sources or different version families are placed into different source record version groups.

[0064] For each source record version group, a version node is established with a version identifier, and the corresponding record and version status are stored in the version node. When the previous version identifier of the current version node is the same as the version identifier of another version node, a directed connection is established from the other version node to the current version node, forming a version succession relationship. The version node with an empty previous version identifier is used as the starting version node. When the same version node connects to multiple subsequent version nodes, multiple version branches are formed.

[0065] If the preceding version identifier is not empty but there is no corresponding version node in the source record version group, the current version node and subsequent version nodes are marked as having a missing preceding version, and the current valid record is not determined using the corresponding version branch; if a circular connection occurs in the version succession relationship, the version nodes involved in the circular connection are marked as having an abnormal version relationship; a version node that is not referenced by other version nodes as a preceding version is designated as the final version node, and the final version node is determined separately for each version branch.

[0066] The preset version status processing rules include version status categories and corresponding processing methods. When the last version node is in the current state, the record corresponding to the last version node is selected. When the last version node is in the historical state, the corresponding version branch is marked as missing subsequent versions, and no currently valid record is generated. When the last version node is in the deleted or revoked state, the query of the corresponding version branch is stopped, and no currently valid record is generated. When the last version node is in the verification failure or generation failure state, the query is performed level by level according to the previous version identifier, skipping version nodes that are consecutively in the verification failure or generation failure state, and the record corresponding to the first version node in the current or historical state is determined as the currently valid record. When encountering a deleted state, revoked state, missing previous version, or no matching version node after reaching the starting version node during the query process, no currently valid record is generated for the corresponding version branch.

[0067] Then, based on the data source identifier and business event type, the preset field conversion table is read, and the source field names in each currently valid record are converted into unified field names. The corresponding field values ​​are also converted according to the unified format in the table. Records without configured field correspondence are retained in the original data source records and do not participate in cross-source field conflict judgment. After the field name and field value are converted, the non-empty field values ​​of the same-named records in different currently valid records are compared. If at least two non-empty field values ​​are different, the corresponding record field is identified as a conflicting field. If the non-empty field values ​​are the same or only one record has a non-empty field value, the corresponding record field is not identified as a conflicting field.

[0068] This example reconstructs the record version path by establishing the relationship between version identifiers, and determines the currently valid records from each version branch by combining version status processing rules. This reduces the number of deleted versions, undoing versions, failed versions, and missing records from previous versions that are involved in field fusion. On this basis, it unifies the field names and field value formats of different data sources, reducing conflicts and misjudgments caused by differences in field naming or data formats.

[0069] As an example, the step of fusing record fields other than the conflicting fields in the existing valid records, establishing conflict value branches for the conflicting fields, and obtaining the fused event record includes: extracting version identifiers from the record version information of each of the existing valid records, and identifying record fields that have completed field alignment and field value format conversion and do not belong to the conflicting fields as non-conflicting fields; storing the non-empty field values ​​of each of the non-conflicting fields after deduplication, and associating the data source identifier and version identifier that provide the field values ​​with the corresponding field values; establishing conflict value branches for each of the conflicting fields according to different field values, and recording the field values, data source identifiers, and version identifiers in each conflict value branch; associating the business object identifier, the business event type, the data source identifier, version identifier, and correction time interval corresponding to each of the existing valid records, the non-conflicting fields, and the conflict value branches to obtain the fused event record.

[0070] It should be noted that a non-empty field value refers to a field value that is not empty, an empty string, or has a missing marker; field source information refers to the data source identifier and version identifier associated with the field value; time source information refers to the data source identifier, version identifier, and correction time interval associated with the current valid record; conflict value branches can be a hierarchical storage structure with conflict field names as parent nodes and different field values ​​as child nodes; and fused event records can be structured records that include event basic information, time source information, non-conflicting fields, and conflict value branches.

[0071] Understandably, the process begins by iterating through the currently valid records in the event record group, extracting the version identifier from the record version information of each currently valid record, and reading the corresponding data source identifier and correction time interval. Following the unified field name, conflicting fields are queried from the record fields that have completed field alignment and field value format conversion. Record fields that are not identified as conflicting fields are then grouped into the non-conflicting field set. When a field value is null, an empty string, or a missing marker, the corresponding record field is not allowed to enter the field fusion process.

[0072] For each non-conflicting field, a unified field name and a unified field value format are used as the basis for deduplication. When the corresponding field value is read for the first time, the field value is stored and associated with the data source identifier and version identifier that provided the field value. When the same field value is read subsequently, the field value is not stored repeatedly, and only the data source identifier and version identifier that have not yet been associated are added. When the same data source identifier and version identifier have already been associated, the field source information is not written repeatedly.

[0073] Then, a parent node is created for each conflicting field, and conflict value branches are created according to different field values. When a field value is read for the first time, a field value branch is created under the corresponding parent node, and the field value, data source identifier, and version identifier are recorded. When the same field value is read subsequently, the corresponding data source identifier and version identifier are added to the existing field value branch. Different field values ​​are kept in different conflict value branches, and field values ​​are not overwritten according to the data source order or the record writing order.

[0074] For each currently valid record, the data source identifier, version identifier, and the start and end times of the correction time interval are associated as time source information. Both field source information and time source information contain data source identifier and version identifier. By using the combination relationship of the two identifiers, the currently valid record and corresponding correction time interval for that field value can be located from the field value.

[0075] Finally, the business object identifier and business event type corresponding to the event record group are read, and the business object identifier, business event type, time source information of each currently valid record, deduplicated non-conflicting field name and field value, field source information, and conflicting field name and corresponding conflicting value branch are associated and stored to form a fused event record; when there is no conflicting field or no non-conflicting field with a non-empty field value, the corresponding content is empty.

[0076] This example reduces duplicate storage of the same field values ​​by deduplicating field values ​​and retains the correspondence between field values ​​and source records by using data source identifiers and version identifiers. It establishes conflict value branches for conflicting fields to avoid different field values ​​from overwriting each other, and writes the correction time interval of each currently valid record into the fusion event record, thereby improving the source traceability capability of the fusion result.

[0077] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the heterogeneous record fusion method based on dual timestamps of this application. Step S20 of the heterogeneous record fusion method based on dual timestamps includes steps S21 to S25: Step S21: Distinguish between reference data source records and non-reference data source records according to the preset reference data source, and read the anchor event type and anchor field mapping relationship representing the same event point in the anchor mapping table; Step S22: Extract the anchor field values ​​of cross-source records according to the anchor field mapping relationship to obtain the reference anchor value combination and the non-reference anchor value combination; Step S23: Pair cross-source records with the same business object identifier, business event type corresponding to the anchor event type, and the same anchor value combination, and each with a unique corresponding record, to obtain time anchor pairs; Step S24: First, group the time anchor pairs according to the data source identifier of the non-reference data source record, and then, according to the event timestamp of the non-reference data source record in each time anchor pair, classify each time anchor pair into the corresponding preset statistical time window. Step S25: Set the clock offset of the preset reference data source to zero, and set the median of the difference between the event timestamps of the non-reference data source records and the reference data source records in each partitioning result as the clock offset.

[0078] It should be noted that "same event time point" refers to event timestamps in records from different data sources all corresponding to the same processing node of the same business event instance, such as all corresponding to the event generation time or all corresponding to the event processing completion time. Anchor event type refers to a business event type that can correspond to the same event time point in different data sources; anchor field mapping relationship refers to the correspondence between fields in the reference data source and fields in the non-reference data source; reference anchor value combination refers to the ordered set of field values ​​extracted from the reference data source record according to the anchor field arrangement order; non-reference anchor value combination refers to the ordered set of field values ​​extracted from the non-reference data source record according to the same arrangement order.

[0079] Understandably, firstly, from the heterogeneous data systems that have been connected, a data source that stores the time service identifier and the most recent synchronization time, and whose synchronization error is no greater than the preset synchronization error upper limit, is selected as the reference data source; the preset synchronization error upper limit is set according to the technical parameters of the corresponding time service, and if the above conditions are not met, the corresponding data source is not used as the time comparison benchmark.

[0080] The anchor mapping table is set up according to the combination of reference data source and non-reference data source. Each mapping record includes reference data source identifier, non-reference data source identifier, anchor event type, reference anchor field name, non-reference anchor field name, and field sequence number. The event timestamp field corresponding to the anchor event type needs to represent the same processing node of the same business event instance. For example, both should represent the event generation time. The event generation time and the event processing completion time cannot be configured as the same anchor event type.

[0081] Then, based on the data source identifier and business event type, the corresponding anchor field mapping relationship is read. The anchor field values ​​in the reference data source record and the non-reference data source record are extracted according to the field sorting number. The reference anchor value combination and the non-reference anchor value combination are generated in the form of an ordered set of field values. The anchor field can be a cross-source request identifier, transaction identifier, batch identifier or event sequence identifier. If any anchor field is missing or the field value is empty, the corresponding record is marked as having an incomplete anchor field and no anchor value combination is generated.

[0082] In the current batch of reference data source records and non-reference data source records, cross-source records with the same business object identifier, business event type corresponding to the same anchor event type, and identical reference anchor value combinations and non-reference anchor value combinations item by item are selected; when the same anchor value combination corresponds to only one record in both the reference data source and the corresponding non-reference data source, the two records are combined into a time anchor pair; when duplicate anchor value combinations exist in either data source, time anchor pairs are not generated using the corresponding records to avoid selecting time correspondences among duplicate records.

[0083] First, time anchor pairs are grouped according to the data source identifier of the non-reference data source records. Then, for each group of time anchor pairs, the corresponding time anchor pairs are assigned to the corresponding preset statistical time window based on the event timestamps of the non-reference data source records in each time anchor pair. The length of the statistical time window is set according to the clock drift change period and the generation frequency of time anchor pairs, and can be from 30 minutes to 24 hours; in this example, it is set to 1 hour. Each statistical time window needs to contain a preset minimum number of anchors, which can be from 5 to 30; in this example, it is set to 10.

[0084] Finally, the clock offset of the reference data source is set to 0. The difference between the event timestamps recorded in the non-reference data source and the event timestamps recorded in the reference data source within each statistical time window is calculated, and the median of the difference is used as the clock offset of the corresponding non-reference data source. When the number of time anchor points within the statistical time window is less than the preset minimum number of anchor points, the current statistical time window is merged with the statistical time windows that are adjacent in time and have the same non-reference data source identifier. The length of the merged window does not exceed 24 hours. When the preset minimum number of anchor points is still not met after the maximum window length is reached, the corresponding data source and time range are marked as insufficient time anchor points, and the corresponding clock offset is not output.

[0085] This embodiment filters cross-source records corresponding to the same event time point by combining anchor event type and anchor field value, excluding records with missing anchor fields or duplicate anchor values, and uses the median of the timestamp difference of multiple time anchor pairs within the same statistical time window to determine the clock offset, thereby reducing the impact of record mismatch and single abnormal timestamps on the clock offset.

[0086] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the heterogeneous record fusion method based on dual timestamps in this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0087] This application also provides a heterogeneous record fusion device based on dual timestamps, please refer to... Figure 3 The heterogeneous record fusion device based on dual timestamps includes: Data acquisition module 10 is used to acquire multi-source heterogeneous record data. The heterogeneous record data includes data source identifier, business object identifier, business event type, dual timestamp, record version information and record content. The dual timestamp consists of an event timestamp and a write timestamp generated by the same data source clock. Anchor pairing module 20 is used to generate time anchor pairs based on a preset reference data source and anchor mapping table, and to determine the clock offset based on the difference in timestamps of cross-source events. The time correction module 30 is used to determine the write latency based on the difference between the two timestamps from the same source, and to generate a correction time interval based on the clock offset and the write latency. Event association module 40 is used to perform cross-source association on the heterogeneous record data according to the business object identifier, the business event type, the record content and the correction time interval to obtain an event record group; Version processing module 50 is used to construct the version succession relationship within the event record group based on the record version information, and to determine the currently valid records and conflict fields; The record fusion module 60 is used to fuse record fields other than the conflicting field in the existing valid records, establish conflict value branches of the conflicting field, and obtain fused event records.

[0088] The heterogeneous record fusion apparatus based on dual timestamps provided in this application, employing the heterogeneous record fusion method based on dual timestamps in the above embodiments, can solve the technical problem of how to associate and fuse cross-source records corresponding to the same event when there are clock skews, write delays, and version evolutions in multi-source heterogeneous records. Compared with the prior art, the beneficial effects of the heterogeneous record fusion apparatus based on dual timestamps provided in this application are the same as those of the heterogeneous record fusion method based on dual timestamps provided in the above embodiments, and other technical features in the heterogeneous record fusion apparatus based on dual timestamps are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0089] This application provides a heterogeneous record fusion device based on dual timestamps. The heterogeneous record fusion device based on dual timestamps includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the heterogeneous record fusion method based on dual timestamps in the first embodiment described above.

[0090] The following is for reference. Figure 4 This document illustrates a structural schematic diagram of a heterogeneous recording fusion device based on dual timestamps suitable for implementing embodiments of this application. The heterogeneous recording fusion device based on dual timestamps in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Android Devices), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The heterogeneous record fusion device based on dual timestamps shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0091] like Figure 4As shown, the heterogeneous recording fusion device based on dual timestamps may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 1002 (Read Only Memory) or a program loaded from storage device 1003 into RAM 1004 (Random Access Memory). RAM 1004 also stores various programs and data required for the operation of the heterogeneous recording fusion device based on dual timestamps. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. I / O interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the dual-timestamp-based heterogeneous recording fusion device to wirelessly or wiredly communicate with other devices to exchange data. Although a dual-timestamp-based heterogeneous recording fusion device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0092] The heterogeneous record fusion device based on dual timestamps provided in this application, employing the heterogeneous record fusion method based on dual timestamps in the above embodiments, can solve the technical problem of how to associate and fuse cross-source records corresponding to the same event when there are clock skews, write delays, and version evolutions in multi-source heterogeneous records. Compared with the prior art, the beneficial effects of the heterogeneous record fusion device based on dual timestamps provided in this application are the same as those of the heterogeneous record fusion method based on dual timestamps provided in the above embodiments, and other technical features in this heterogeneous record fusion device based on dual timestamps are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0093] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the heterogeneous record fusion method based on dual timestamps in the above embodiments.

[0094] The computer-readable storage medium provided in this application can be a tangible medium capable of storing computer programs, such as a hard disk, random access memory, read-only memory, flash memory, optical storage device, or magnetic storage device. When the computer program is executed by a processor, it implements the above-described heterogeneous record fusion method based on dual timestamps.

[0095] The aforementioned computer-readable storage medium may be included in a heterogeneous recording fusion device based on dual timestamps; or it may exist independently and not assembled into a heterogeneous recording fusion device based on dual timestamps.

[0096] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a heterogeneous record fusion device based on dual timestamps, the device performs the following actions: acquires multi-source heterogeneous record data, which includes a data source identifier, a business object identifier, a business event type, dual timestamps, record version information, and record content. The dual timestamps consist of an event timestamp generated by a clock from the same data source and a write timestamp. It then generates time anchor pairs based on a preset reference data source and an anchor mapping table, and determines a clock offset based on the cross-source event timestamp difference. It determines a write latency based on the same-source dual timestamp difference, and generates a correction time interval based on the clock offset and the write latency. It performs cross-source association of the heterogeneous record data based on the business object identifier, the business event type, the record content, and the correction time interval to obtain an event record group. It constructs a version succession relationship within the event record group based on the record version information, and determines currently valid records and conflict fields. Finally, it merges record fields from the currently valid records, excluding the conflict fields, and establishes conflict value branches for the conflict fields to obtain fused event records.

[0097] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0098] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described heterogeneous record fusion method based on dual timestamps. This solves the technical problem of how to associate and fuse cross-source records corresponding to the same event when there are clock skews, write delays, and version evolutions in multi-source heterogeneous records. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the heterogeneous record fusion method based on dual timestamps provided in the above embodiments, and will not be repeated here.

[0099] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A heterogeneous record fusion method based on dual timestamps, characterized in that, The method includes: Acquire multi-source heterogeneous record data, which includes data source identifier, business object identifier, business event type, dual timestamp, record version information, and record content. The dual timestamp consists of an event timestamp and a write timestamp generated by the same data source clock. Time anchor pairs are generated based on the preset reference data source and anchor mapping table, and the clock offset is determined based on the difference in timestamps of cross-source events. The write latency is determined based on the difference between the two timestamps from the same source, and a correction time interval is generated based on the clock offset and the write latency. Based on the business object identifier, the business event type, the record content, and the correction time interval, the heterogeneous record data is cross-sourced to obtain an event record group; Based on the record version information, construct the version succession relationship within the event record group, and determine the currently valid records and conflict fields; Merge all record fields in the existing valid records except for the conflicting field, establish the conflict value branch of the conflicting field, and obtain the merged event record; The steps of generating time anchor pairs based on a preset reference data source and anchor mapping table, and determining the clock offset based on the cross-source event timestamp difference, include: Based on the preset reference data source, distinguish between reference data source records and non-reference data source records, and read the anchor event type and anchor field mapping relationship representing the same event point in the anchor mapping table; The anchor field values ​​of cross-source records are extracted according to the anchor field mapping relationship to obtain the combination of reference anchor values ​​and the combination of non-reference anchor values; Pair cross-source records with the same business object identifier, business event type corresponding to the anchor event type, and the same combination of anchor values, and each with a unique corresponding record, to obtain time anchor pairs; First, the time anchor pairs are grouped according to the data source identifier of the non-reference data source record. Then, based on the event timestamp of the non-reference data source record in each time anchor pair, each time anchor pair is assigned to the corresponding preset statistical time window. The clock offset of the preset reference data source is set to zero, and the median of the difference between the event timestamps of the non-reference data source records and the reference data source records in each partition result is determined as the clock offset. The step of determining the write latency based on the difference between the two timestamps from the same source, and generating a correction time interval based on the clock offset and the write latency, includes: For each record, the first correction time is obtained by subtracting the corresponding clock offset from the event timestamp; The heterogeneous record data is divided according to the data source identifier, business event type, and the preset statistical time window to which the first correction time belongs; Calculate the difference between the write timestamp and the event timestamp in each partition result, and determine the median of the obtained differences as the write latency. Subtract the corresponding clock offset and write delay from the write timestamp of each record to obtain the second correction time; The preset time tolerance is subtracted from the earlier of the first and second correction times, and the preset time tolerance is added to the later time to obtain the correction time interval.

2. The method as described in claim 1, characterized in that, The step of performing cross-source association on the heterogeneous record data based on the business object identifier, the business event type, the record content, and the correction time interval to obtain the event record group includes: Reference data source records and non-reference data source records are distinguished according to a preset reference data source, and reference data source records with the same business object identifier and business event type and intersecting correction time intervals are paired with non-reference data source records to form candidate record pairs; Extract the common comparison fields of the candidate record pairs according to the preset comparison field table, and unify the field value format; When the common comparison field is not empty, the ratio of the number of common comparison fields with the same field value to the total number of common comparison fields is determined as the content matching degree; For each non-reference data source record, when the candidate record pair with the highest content matching degree is unique and the maximum content matching degree is not less than the preset matching degree threshold, an event association is established between the records within the candidate record pair. Group the records of the same reference data source and their corresponding associated non-reference data source records into an event record group.

3. The method as described in claim 1, characterized in that, The step of constructing the version succession relationship within the event record group based on the record version information, and determining the currently valid records and conflict fields, includes: Extract the version family identifier, version identifier, previous version identifier, and version status from the record version information, and divide the records in the event record group according to the data source identifier and the version family identifier to obtain the source record version group; The version identifier recorded in the source record version group is used as the version node, and the version node is connected according to the previous version identifier to obtain the version succession relationship; Version nodes that are not referenced by other version nodes as preceding versions are identified as end version nodes. Starting from the end version node, the current valid record is determined by querying along the preceding version identifier according to the preset version status processing rules. Align the record fields in the currently valid records according to the preset field conversion table, and convert the field values ​​to the preset uniform format; When the non-empty field values ​​of the same-named record field in at least two existing valid records are different, the same-named record field is identified as a conflicting field.

4. The method as described in claim 1, characterized in that, The steps of fusing record fields other than the conflicting field from the existing valid records, establishing conflict value branches for the conflicting fields, and obtaining the fused event records include: Extract version identifiers from the record version information of each of the existing valid records, and determine the record fields that have completed field alignment and field value format conversion and do not belong to the conflicting fields as non-conflicting fields; After deduplicating the non-empty field values ​​of each of the non-conflicting fields, store them and associate the data source identifier and version identifier that provide the field value with the corresponding field value; For each of the conflicting fields, a conflict value branch is established according to different field values, and the field value, data source identifier, and version identifier are recorded in each conflict value branch; By associating the business object identifier, the business event type, the data source identifier, version identifier, and correction time interval corresponding to each of the currently valid records, the non-conflicting fields, and the conflicting value branches, a fused event record is obtained.

5. The method according to any one of claims 1 to 4, characterized in that, The steps for acquiring multi-source heterogeneous record data include: Receive source records from various heterogeneous data systems and configure data source identifiers; Based on the preset object identifier mapping table and the preset event type mapping table, the source object identifier and source event type in the source record are respectively converted into business object identifier and business event type; Extract the event timestamp and write timestamp generated by the same data source clock from the source record to obtain a double timestamp; Extract the version family identifier, version identifier, previous version identifier and version status from the source record to obtain the record version information, and encapsulate the business fields and field values ​​into record content; By associating the data source identifier, the business object identifier, the business event type, the dual timestamps, the record version information, and the record content, heterogeneous record data is obtained.

6. A heterogeneous record fusion device based on dual timestamps, characterized in that, The apparatus is used to perform the heterogeneous record fusion method based on dual timestamps as described in any one of claims 1 to 5, the apparatus comprising: The data acquisition module is used to acquire multi-source heterogeneous record data. The heterogeneous record data includes data source identifier, business object identifier, business event type, dual timestamp, record version information and record content. The dual timestamp consists of an event timestamp and a write timestamp generated by the same data source clock. The anchor pairing module is used to generate time anchor pairs based on a preset reference data source and anchor mapping table, and to determine the clock offset based on the difference in timestamps of cross-source events. The time correction module is used to determine the write latency based on the difference between the two timestamps from the same source, and to generate a correction time interval based on the clock offset and the write latency. The event association module is used to perform cross-source association of the heterogeneous record data based on the business object identifier, the business event type, the record content, and the correction time interval to obtain an event record group; The version processing module is used to construct the version succession relationship within the event record group based on the record version information, and to determine the currently valid records and conflict fields; The record fusion module is used to fuse record fields other than the conflicting field in the existing valid records, establish conflict value branches of the conflicting field, and obtain fused event records.

7. A heterogeneous recording fusion device based on dual timestamps, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the heterogeneous record fusion method based on dual timestamps as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the heterogeneous record fusion method based on dual timestamps as described in any one of claims 1 to 5.

Images