A heterogeneous data integration and processing method and system

Abstract

This invention discloses a heterogeneous data integration and processing method and system, relating to the field of data processing and data integration technology. The method includes the following steps: collecting continuous operation records corresponding to the heterogeneous data processed by each node during operation; organizing the continuous operation records according to a unified chronological order; and writing node status markers into the organized continuous operation records to form a continuous operation sequence. This invention identifies parallel segments formed during network outages by constructing a continuous operation sequence and backtracks and organizes duplicate segments to generate a unique trajectory sequence, thereby eliminating the superposition of duplicate records and ensuring that the data statistics reflect the actual data progression relationship. Simultaneously, by extracting the actual growth segments in business volume changes as the basis for capacity expansion triggering, the resource allocation rhythm is adjusted to ensure that resource scheduling behavior is consistent with business growth, thus maintaining the stable progress of the overall operation process.

Description

Technical Field

[0001] This invention relates to the field of data processing and data integration technology, specifically to a method and system for heterogeneous data integration and processing. Background Technology

[0002] Heterogeneous data integration and processing refers to the technical process of integrating, transforming, and coordinating data resources from different sources, with different structures, formats, and semantic standards, under unified rules, so that they can be associated, analyzed, and utilized within the same system. Heterogeneity is typically reflected in data storage formats (such as relational databases, log files, and sensor stream data), data representation formats (structured, semi-structured, and unstructured), encoding standards, and inconsistent field semantic definitions. Integration focuses on resolving issues related to structural mapping, field alignment, semantic disambiguation, and unified labeling between data. Processing includes data cleaning, format conversion, conflict resolution, time synchronization, association matching, and result output. Through this type of technology, previously scattered and inconsistent data resources can be integrated into a collaboratively analyzeable data system, providing a unified data foundation for subsequent data analysis, decision support, or intelligent applications.

[0003] The existing technology has the following shortcomings: In existing technologies, heterogeneous data integration systems often employ distributed node parallel processing mechanisms to improve overall throughput. However, when a short-term network disconnection occurs between nodes, each node continues to operate independently according to its local state, unable to obtain real-time information about the data processing progress of other nodes. When data is aggregated after network recovery, the lack of a unified verification mechanism can easily lead to duplicate records, causing abnormally amplified system statistics and being misjudged as a surge in business volume. In this situation, automatic scaling strategies are easily triggered incorrectly, rapidly consuming a large amount of computing resources and even risking resource pool exhaustion. For example, in real-time log aggregation scenarios on cloud computing platforms, if duplicate writes are not identified after a short-term link interruption and synchronization is restored, it can lead to an abnormally high platform load index, affecting normal business scheduling and system stability.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for heterogeneous data integration and processing to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for heterogeneous data integration and processing, comprising the following steps: Collect the continuous operation records corresponding to the heterogeneous data processed by each node during operation, organize the continuous operation records according to a unified time order, and write node status flags into the organized continuous operation records to form a continuous operation sequence. Based on the continuous running sequence, the data advancement rhythm of each node is compared. Parallel segments formed by the independent advancement of each node during the network interruption are extracted from the continuous running sequence. The positions of repeated business operations are marked in the parallel segments to generate a list of repeated segments. Based on the list of repeated segments, trace back the continuous operation sequence, extract the original operation trajectory in the corresponding segment of the list of repeated segments, rearrange the original operation trajectory according to the order of business source, and generate a unique trajectory sequence. The continuous running sequence is rearranged based on the unique trajectory sequence. The unique trajectory sequence is used to replace the corresponding segment of the duplicate segment list in the continuous running sequence, and a deduplicated unified sequence is generated while maintaining the unified time progression order. Based on the deduplicated unified sequence, the business volume change sequence is reorganized, and the real growth segment is extracted from the business volume change sequence. The real growth segment is used as the basis for expansion triggering to adjust the resource allocation rhythm, forming a stable operation process.

[0007] Preferably, the continuous sequence formation steps are as follows: Collect operation records corresponding to data reading, data parsing, data conversion, data writing, and data forwarding actions generated during the processing of heterogeneous data at each node. Write node identification information, business source identification information, data source identification information, and operation occurrence time information into the operation records to form a continuous operation record set. Read the operation occurrence time information from the continuous operation record set, and organize the operation records in the continuous operation record set according to a unified time scale to form a unified time progression structure; Read the node running status information in the unified time advancement structure, write the node status flags corresponding to the normal operation status, network interruption status and network recovery status into the operation record, and form an operation record containing node status flags; The operation records containing node status markers are arranged continuously according to a unified time progression structure to form a continuous operation sequence that unfolds in the order of time progression.

[0008] Preferably, the steps for generating the list of repeating fragments are as follows: Read the node identification information, business source identification information, data source identification information, operation occurrence time information, and node status markers in the continuous running sequence, scan and organize the continuous running sequence according to a unified time progression order, and form a node running trajectory comparison structure on a unified time axis; Based on the node running trajectory comparison structure, the network interruption status information in the node status marker is read, the network interruption section is located on a unified time axis, and the parallel segments formed by the independent advancement of each node in the network interruption section are extracted from the continuous running sequence. Read the business source identification information and operation occurrence time information in the parallel segment, arrange the operation records in the parallel segment according to the business source relationship, and mark the position of duplicate business operation at the position corresponding to the same business source identification information; Read the time segment information corresponding to the parallel segments, the node operation trajectory information, and the location marking information of repeated business operations, and record them centrally in the order of time progression to form a list of repeated segments.

[0009] Preferably, the list of repeated segments records time segment information, node operation trajectory information, and repeated business operation location marking information, and is arranged continuously in the order of time progression. Within each time segment, the correspondence between node operation trajectory and business source identification information is maintained, forming a data record structure for describing the independent progress of each node during network interruption.

[0010] The preferred steps for generating a unique trajectory sequence are as follows: Based on the time segment information in the list of repeated segments, operation records containing node identifier information, business source identifier information, data source identifier information, operation occurrence time information, and node status markers are extracted from the corresponding segments in the continuous operation sequence to form the original operation trajectory set; Read the business source identification information from the original operation trajectory set, merge and organize the operation records in the original operation trajectory set according to the business source identification information, and keep the arrangement relationship between the operation occurrence time information and the node identification information unchanged to form a business source advancement trajectory; The operation records are arranged sequentially based on the business source identification information in the business source advancement trajectory, and the operation occurrence time sequence is continuously unfolded within the business source advancement trajectory to form a unified business advancement sequence; The operation records are arranged continuously according to the unified business progress order, and a unique trajectory sequence is generated while maintaining the consistency of node identification information, business source identification information and operation occurrence time information.

[0011] Preferably, the operation records in the unified business advancement sequence are arranged continuously according to the business source identification information. The operation records corresponding to the same business source identification information form a continuous advancement relationship according to the operation occurrence time information. The operation records corresponding to different business source identification information are arranged according to the order of the business source identification information, forming a unified business advancement structure in a unique trajectory sequence.

[0012] The preferred steps for generating the deduplicated uniform sequence are as follows: Based on the time segment information in the list of repeated segments, locate the corresponding segment in the continuous running sequence, and extract the operation records containing node identification information, business source identification information, data source identification information, operation occurrence time information, and node status markers to form a segment operation record set; Read the business source identifier information, operation occurrence time information, and node identifier information from the unique trajectory sequence, expand and arrange the operation records in the unique trajectory sequence according to the order of business source, and map them to the corresponding set of segment operation records; Replace the set of segment operation records for the corresponding time segment in the continuous running sequence with the operation records in the unique trajectory sequence, and insert the operation records from the unique trajectory sequence into the corresponding time segment; Read the operation time information in the continuous running sequence, arrange the continuous running sequence after the replacement is completed in a unified time progression order, and generate a unified sequence after deduplication.

[0013] Preferably, the steps for generating the business volume change sequence and adjusting the resource allocation rhythm are as follows: Read the operation records in the deduplicated unified sequence, establish a unified time scale statistical structure based on the operation occurrence time information, count the number of business processing records at the corresponding time scale position in the unified time scale statistical structure, and form a business volume change sequence. Read the number of business processing records in the business volume change sequence, compare and organize the number of business processing records at adjacent time scale positions, and divide the business volume change sequence into business growth segment, business stability segment, and business decline segment. Read the time scale position and the number of business processing records corresponding to the business growth segment, extract the records of the business growth segment, and form a set of real growth segments; Based on the time scale position and the number of business processing records in the set of actual growth segments, the resource allocation rhythm is adjusted to form a stable operation process under the condition of unified time advancement sequence.

[0014] A heterogeneous data integration and processing system includes a runtime record construction module, a parallel segment recognition module, a trajectory generation module, a unified sequence reconstruction module, and a resource scheduling and control module. The operation record construction module collects the continuous operation records corresponding to the heterogeneous data processed by each node during operation, organizes the continuous operation records according to a unified time order, and writes node status markers into the organized continuous operation records to form a continuous operation sequence. The parallel segment identification module compares the data advancement rhythm of each node based on the continuous running sequence, extracts the parallel segments formed by the independent advancement of each node during the network interruption in the continuous running sequence, marks the positions of repeated business operations in the parallel segments, and generates a list of repeated segments. The trajectory generation module traces back the continuous running sequence based on the list of repeated segments, extracts the original operation trajectory in the corresponding segment of the list of repeated segments, rearranges the original operation trajectory according to the order of business source, and generates a unique trajectory sequence. The unified sequence reconstruction module rearranges the continuous running sequence based on the unique trajectory sequence. It replaces the corresponding segments of the duplicate segment list in the continuous running sequence with the unique trajectory sequence, and generates a deduplicated unified sequence while maintaining the unified time progression order. The resource scheduling and control module reorganizes the business volume change sequence based on the deduplicated unified sequence, extracts the real growth segment from the business volume change sequence, and uses the real growth segment as the basis for expansion triggering to adjust the resource allocation rhythm, forming a stable operation process.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention constructs a continuous operation sequence by uniformly organizing the continuous operation records generated during the operation of each node in a chronological order and introducing node status markers into the organized records. Based on this continuous operation sequence, parallel segments formed during network outages are identified, and the locations of repetitive business operations are further located. By backtracking and organizing repetitive segments to form a unique trajectory sequence, the multi-node parallel advancement records originally formed during network fluctuations can be replaced by a unified data advancement trajectory. This eliminates duplicate records in the data processing process, allowing the data statistics process to be organized based on the actual data advancement relationship and avoiding abnormal amplification of business volume statistics results due to the superposition of duplicate records.

[0016] This invention reorganizes the business volume change sequence based on a deduplicated unified sequence, extracts the actual growth segments from the business volume change sequence, and uses these actual growth segments as the basis for adjusting the resource allocation rhythm to trigger capacity expansion. This ensures that the resource input process progresses gradually according to the actual business growth trajectory, thereby maintaining consistency between resource scheduling behavior and changes in business processing scale. Through this method, resource allocation behavior during operation can be controlled based on actual business changes, ensuring a stable overall operation and preventing the impact of abnormal business volume changes caused by duplicate recordings on resource scheduling. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart of a heterogeneous data integration and processing method according to the present invention.

[0019] Figure 2 This is a schematic diagram of a heterogeneous data integration and processing system according to the present invention.

[0020] Figure 3 This is a mind diagram for the present invention. Detailed Implementation

[0021] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0022] This invention provides, for example Figure 1 The method for integrating and processing heterogeneous data, as shown, includes the following steps: Collect the continuous operation records corresponding to the heterogeneous data processed by each node during operation, organize the continuous operation records according to a unified time order, and write node status flags into the organized continuous operation records to form a continuous operation sequence. In a distributed data processing environment, each node continuously generates operation records reflecting its processing behavior and operational status during the execution of data processing tasks. By continuously organizing these operation records, a complete data progression trajectory can be formed, thus providing a continuous data foundation for subsequent analysis of data progression during network fluctuation phases. The specific implementation steps are as follows: During the processing of heterogeneous data at each node, the data processing actions performed by the nodes are continuously recorded. Upon receiving heterogeneous data, each node sequentially performs data reading, data parsing, data transformation, data writing, and data forwarding actions, generating a corresponding operation record for each processing action. When generating the operation record, node identification information, business source identification information, data source identification information, and operation occurrence time information are written into the record. The node identification information indicates the node from which the processing action was performed; the business source identification information indicates the business type of the data; the data source identification information indicates the data input source corresponding to the data entering the node; and the operation occurrence time information indicates the specific time when the processing action occurred. Simultaneously, the execution order of each processing action is recorded, allowing the operation records to reflect the progression of data processing within the node. By continuously recording all processing actions generated during the processing of heterogeneous data, a continuous set of operation records with node identification information, business source identification information, data source identification information, and operation occurrence time information is formed, thus enabling the continuous operation record set to fully present the entire processing trajectory of each node in processing heterogeneous data.

[0023] After constructing the continuous operation record set, all operation records in the set are organized according to a unified chronological order. By reading the operation occurrence time information in each operation record, operation records from different nodes are arranged in chronological order. During the arrangement process, the operation occurrence time information is aligned to a unified time scale, enabling operation records generated by different nodes under different time environments to be arranged within a unified time scale system. The unified time scale system uses a fixed time unit as the time advancement benchmark, mapping the time information of all operation records to a unified time scale position, allowing operation records generated by each node to be arranged on the same time axis. After completing the time scale alignment, all operation records are organized one by one according to chronological order, arranging the operation records sequentially on the time axis to form a unified time advancement structure. In this unified time advancement structure, each operation record retains its original node identification information, business source identification information, and data source identification information without change, enabling the unified time advancement structure to accurately represent the operation trajectory generated by different nodes processing heterogeneous data.

[0024] After unifying the time sequence, node status markers are written into the organized operation records. By reading the node's operational status information at the corresponding time point in the operation record, the node's operational status when performing the corresponding processing action is written into the corresponding operation record. Node operational status information includes three categories: normal operation status, network interruption status, and network recovery status. Normal operation status indicates that the node performs data processing actions in a stable communication environment; network interruption status indicates that the node continues to perform local data processing actions when the communication link is interrupted; and network recovery status indicates that the node performs data synchronization actions after the communication link is restored. By writing node operational status information into the corresponding operation records, each operation record simultaneously includes node identification information, business source identification information, data source identification information, operation occurrence time information, and node status markers. Through this method, the organized operation records not only reflect the time sequence of processing actions but also reflect the data progress of the node in different operational stages, thus providing both time-dimensional and operational status-dimensional information.

[0025] After writing the node status markers, the operation records containing these markers are concatenated and organized. Following the chronological order of the unified time-progression structure, all operation records are connected one by one to form a continuous arrangement. During this arrangement, the chronological order, node identification information, and business source identification information between the operation records remain unchanged, ensuring a continuous progression relationship between them. After this overall concatenation and organization, a complete continuous execution sequence is formed. In this continuous execution sequence, each operation record is arranged in chronological order. The operation trajectories generated by each node during the processing of heterogeneous data can be continuously unfolded on a unified timeline. Simultaneously, the node status markers contained in each operation record can present the data progression rhythm of the node in different operational stages. This allows the continuous execution sequence to clearly show the data progression of each node during the processing of heterogeneous data during network fluctuations, providing a continuous data foundation for subsequent comparison of node progression rhythms and extraction of parallel segments.

[0026] Based on the continuous running sequence, the data advancement rhythm of each node is compared. Parallel segments formed by the independent advancement of each node during the network interruption are extracted from the continuous running sequence. The positions of repeated business operations are marked in the parallel segments to generate a list of repeated segments. By analyzing the node operation trajectories presented by the continuous operation sequence, the data processing behavior of different nodes in the same time advancement structure can be segmented, thereby identifying independent advancement processes formed during network outages and further locating the positions of repetitive service operations. In this process, by continuously unfolding and segmenting the continuous operation sequence, a list of repetitive segments is generated to record repeated advancements. The specific implementation steps are as follows: The established continuous operation sequence is systematically expanded and organized. This involves reading the node identifier, business source identifier, data source identifier, operation occurrence time, and node status flags contained in each operation record within the sequence, and scanning the sequence line by line according to a unified time progression order. During the scanning process, the operation records in the continuous operation sequence are categorized and arranged according to the node identifier information, ensuring that operation records from the same node form a continuous node operation trajectory along the time progression direction, while maintaining the time order of the operation records in the continuous operation sequence. Subsequently, a node operation trajectory comparison structure is established on a unified time axis. In this comparison structure, operation records are arranged line by line according to the operation occurrence time information, and the content of operation records from different nodes is displayed at each time position. This allows the data processing actions performed by each node at the same time progression position to be compared and displayed at the same time axis position. Through this organization method, the operation records in the continuous operation sequence can form a complete node progression rhythm display structure. This structure can continuously present the data progression trajectory of each node in the process of processing heterogeneous data, and provide a time series basis for subsequent identification of independent progression situations formed during network outages.

[0027] After establishing the node progression rhythm display structure, the node status markers in the continuous running sequence are read one by one, and the time segments during network interruption are identified based on the running status information recorded in the node status markers. When reading node status markers, if an operation record in the continuous running sequence has a node status marker indicating a network interruption state within the same time progression segment, that time segment is identified as a network interruption segment, and all operation records within that time segment are centrally organized. During this organization process, the operation records within the network interruption segment are separated and arranged according to node identification information, so that each node forms an independent data progression trajectory within that time segment, while maintaining the original temporal order of the operation records. Subsequently, the operation records formed by each node within the network interruption segment are continuously arranged, so that the data progression trajectory of each node can fully present all the data processing actions it performed during the communication link interruption. Through the above organization process, parallel segments formed by the independent progression of each node during the network interruption can be extracted from the continuous running sequence. These parallel segments fully record the data progression trajectories formed by multiple nodes performing data processing actions within the same time segment.

[0028] After extracting parallel segments, the operation records within each segment are systematically organized. By reading the business source identifier and data source identifier information from each operation record, the operation records from different nodes within the parallel segment are arranged accordingly. During the arrangement process, operation records with the same business source identifier information are grouped together within the same time frame, maintaining the original chronological order of the operation records. This ensures that the operation records in the parallel segment can form a continuous business progression trajectory based on their business source relationships. When multiple nodes record operation records with the same business source identifier information within the same time frame, the corresponding time positions of these operation records are marked as duplicate business operation positions. Simultaneously, the corresponding node identifier, business source identifier, and operation occurrence time information are recorded, ensuring that each duplicate business operation position fully reflects the trajectory of multiple operation records formed by the same business source data in different nodes. By reading and arranging all operation records in the parallel segment one by one, a clear distribution structure of duplicate business operations can be formed within the parallel segment, thereby determining the positions of duplicate business operations generated by each node during independent progression.

[0029] After marking the locations of repetitive service operations within parallel segments, all marked parallel segments are systematically organized. By reading the time segment information, node identifier information, and distribution of repetitive service operation locations for each parallel segment, each segment is recorded sequentially according to its chronological order. During the recording process, a segment number is assigned to each parallel segment, and the data progression trajectory of all nodes within the parallel segment and the distribution information of repetitive service operation locations are stored in the records. After recording all parallel segments, these records are arranged chronologically to form a complete list of repetitive segments. Each segment entry in the list contains corresponding time segment information, node progression trajectory information, and repetitive service operation location marking information. This allows for a structured recording of the parallel progression during all network outages in the continuous running sequence, providing a clear time segment location basis for subsequent backtracking of the continuous running sequence and extraction of the original operation trajectory based on the repetitive segment list.

[0030] Based on the list of repeated segments, trace back the continuous operation sequence, extract the original operation trajectory in the corresponding segment of the list of repeated segments, rearrange the original operation trajectory according to the order of business source, and generate a unique trajectory sequence. Based on the time segment information and node progression trajectory information recorded in the list of repeated segments, targeted backtracking and organization of continuous operation sequences can be performed. By extracting the original operation trajectory within the corresponding time segment and rearranging the operation records according to the business source order, a unique trajectory sequence that reflects the actual data progression relationship can be formed. The specific implementation steps are as follows: Based on the segment numbers, time segment information, and node progression trajectory information recorded in the duplicate segment list, the continuous running sequence is backtracked and located. By reading the start and end times of the time segments in the duplicate segment list, the range of operation records corresponding to those time segments is searched in the continuous running sequence, and all operation records within that range are extracted. During the extraction process, the node identifier, business source identifier, data source identifier, operation occurrence time, and node status markers in the operation records are completely preserved, ensuring that the extracted operation records accurately reflect the data processing actions performed by each node within the corresponding time segment. Subsequently, the extracted operation records are arranged sequentially in chronological order, allowing the complete data progression trajectory formed by each node within that time segment to be presented, thus forming the original operation trajectory set for the corresponding time segment. Through the above method, all operation records in the continuous running sequence corresponding to the segments in the duplicate segment list can form a complete original operation trajectory foundation.

[0031] After forming the initial set of operation trajectories, the operation records within this set are organized according to their business source relationships. By reading the business source identifier information from each record in the initial set, operation records with the same business source identifier are merged, allowing data processing records from different nodes but belonging to the same business source to form a unified business progression trajectory. During the merging process, the operation occurrence time information and node identifier information in each operation record remain unchanged. Simultaneously, operation records corresponding to the same business source are arranged consecutively according to the order of operation occurrence time, ensuring that the data processing process of that business source forms a complete progression chain. Through this organization process, multiple independent business source progression trajectories can be formed in the initial set of operation trajectories, allowing the data processing processes from different business sources to present their respective data progression sequences. This provides a business source-level arrangement basis for subsequent overall rearrangement of the initial operation trajectories.

[0032] After organizing the business source progression trajectories, the original set of operation trajectories is rearranged as a whole. By arranging the original set of operation trajectories according to the business source identifier information, operation records with the same business source identifier information are kept continuously distributed during the arrangement process, and the chronological order of operations within each business source progression trajectory remains unchanged. Based on maintaining the order within each business source, the progression trajectories of each business source are arranged according to the order of the business source identifier information, so that the operation records in the original set of operation trajectories can form a unified business progression order. Through this arrangement method, the data processing trajectories that were originally proceeding in parallel at different nodes are reorganized into data processing trajectories that proceed continuously according to the order of business sources, enabling the original set of operation trajectories to form a data progression structure that does not contain duplicate progression paths.

[0033] After rearranging the original operation trajectory set, the organized operation records are continuously serialized, maintaining consistency in the operation occurrence time, node identification, and business source identification information to form a complete and unique trajectory sequence. In this unique trajectory sequence, the data processing records corresponding to each business source are arranged sequentially according to the business source order. Simultaneously, within each business source's advancement trajectory, the operation occurrence time order is maintained, ensuring the entire unique trajectory sequence fully reflects the actual advancement relationship during data processing. Through this method, the multi-node parallel advancement trajectories formed during network interruptions in the continuous running sequence are organized into a unique trajectory sequence that advances continuously according to the business source order, thus providing a clear data advancement trajectory foundation for subsequent overall organization of the continuous running sequence and replacement of corresponding segments.

[0034] The continuous running sequence is rearranged based on the unique trajectory sequence. The unique trajectory sequence is used to replace the corresponding segment of the duplicate segment list in the continuous running sequence, and a deduplicated unified sequence is generated while maintaining the unified time progression order. Based on the correspondence between unique trajectory sequences and continuous running sequences, the continuous running sequences can be rearranged in a targeted manner to replace repetitive advancement trajectories formed during network interruptions with unified advancement trajectories, thereby forming a deduplicated unified sequence that accurately reflects the data advancement process. This process is achieved step-by-step through three stages: time segment positioning, trajectory replacement and rearrangement, and overall order preservation. This allows the continuous running sequences to complete trajectory reconstruction while maintaining the stability of the time advancement structure. The specific implementation steps are as follows: Based on the time segment information recorded in the duplicate segment list, the corresponding segments are located segment by segment in the continuous running sequence. By reading the start and end times of the time segment corresponding to each segment number in the duplicate segment list, the range of operation records matching the time segment is searched in the continuous running sequence. During the location process, the operation occurrence time information in the continuous running sequence is scanned one by one, and all operation records whose time falls between the start and end times of the time segment are marked and extracted to form a segment operation record set. When forming the segment operation record set, the node identifier, business source identifier, data source identifier, operation occurrence time information, and node status flag of each operation record are retained, so that the segment operation record set can completely reflect the entire data progression trajectory formed by each node in the process of processing heterogeneous data within the time segment. In this way, all time segments corresponding to the duplicate segment list can be accurately located in the continuous running sequence, and corresponding segment operation record sets can be formed, thus providing a basic data range for subsequent trajectory replacement and organization.

[0035] After extracting the segment operation record set, the segment operation record set is mapped and organized to the unique trajectory sequence corresponding to the time segment. By reading the business source identifier information, operation occurrence time information, and node identifier information from the unique trajectory sequence, the operation records in the unique trajectory sequence are expanded one by one, while maintaining the existing business source order relationship within the unique trajectory sequence. During the organization process, the operation records in the unique trajectory sequence are mapped to the segment operation record set in the continuous running sequence according to the time progression relationship, so that each operation record in the unique trajectory sequence can form a mapping relationship with the corresponding time position in the continuous running sequence. After completing the mapping organization, the operation records in the unique trajectory sequence are arranged sequentially according to the original business source progression order, while maintaining the continuous expansion of the time progression relationship of the operation occurrence time information, so that the unique trajectory sequence can form a complete and unified progression trajectory structure.

[0036] After completing the mapping and organization of the unique trajectory sequence and the segment operation record set, trajectory replacement and organization are performed on the corresponding time segments in the continuous running sequence. This involves removing all operation records corresponding to the segment operation record set from the continuous running sequence and inserting operation records from the unique trajectory sequence into the original time segment positions. This replaces the parallel advancement trajectories in the original segment operation record set with the data advancement trajectories in the unique trajectory sequence. During the insertion process, the operation records in the unique trajectory sequence are inserted sequentially into the corresponding time segment positions according to the order of business origin, while maintaining the original business origin identifier, node identifier, data origin identifier, and operation occurrence time information in the operation records unchanged. Through this method, the original multi-node parallel advancement trajectories in the continuous running sequence are replaced by the unified advancement trajectory formed by the unique trajectory sequence, thereby eliminating duplicate advancement paths formed during network interruption phases.

[0037] After completing the trajectory replacement and organization, the entire continuous operation sequence is organized according to its overall time progression. By reading the operation occurrence time information of all operation records in the continuous operation sequence, the operation records are arranged continuously according to the time progression, ensuring that all operation records maintain a sequential progression relationship on a unified timeline. During the arrangement process, the node identification information, business source identification information, and node status markers in the operation records are kept consistent, so that the continuous operation sequence retains its original operational status information and data source information after the trajectory replacement is completed. By organizing the continuous operation sequence according to its overall time progression, a complete deduplication unified sequence can be formed. In this unified sequence, the data progression trajectories of each business source unfold continuously according to the time progression relationship, and the repeated progression trajectories formed during the network interruption phase have been replaced by a unique trajectory sequence. This allows the organized unified sequence to truly reflect the actual data progression of each node in processing heterogeneous data, providing a stable data foundation for subsequent business volume change organization based on the unified sequence.

[0038] Based on the deduplicated unified sequence, the business volume change sequence is reorganized, and the real growth segment is extracted from the business volume change sequence. The real growth segment is used as the basis for expansion triggering to adjust the resource allocation rhythm, forming a stable operation and advancement process. By analyzing the data progression trajectory presented by the deduplicated unified sequence, the changes in business processing scale over time can be continuously analyzed to form a business volume change sequence. Within this sequence, genuine growth segments can be identified, and resource allocation can be adjusted based on these segments to ensure stable overall operation. The specific implementation steps are as follows: Each operation record in the deduplicated unified sequence is read individually, and a unified time scale statistical structure is established based on the operation occurrence time information in the operation records. By reading the operation records in the deduplicated unified sequence one by one in chronological order, the operation occurrence time corresponding to each operation record is mapped to a unified time scale position, and the business source identifier information and data source identifier information of the corresponding operation record are recorded at that time scale position. After completing the time scale mapping, the number of operation records appearing within each time scale position is counted, and the statistical result is written to the business processing record quantity position corresponding to that time scale, so that each time scale position forms a corresponding business processing record quantity value. Subsequently, all time scale positions are arranged continuously in chronological order, so that the number of business processing records corresponding to each time scale position forms a continuous statistical structure in chronological order, thereby obtaining a complete business volume change sequence. Through the above processing method, the operation records in the deduplicated unified sequence can be transformed into a business processing scale change trajectory arranged continuously in chronological order.

[0039] After generating the business volume change sequence, the changes in the number of business processing records within the sequence are analyzed segment by segment. By reading the number of business processing records sequentially according to the time scale, the number of records at adjacent time scale positions is continuously compared. When the number of records at a later time scale position is higher than that at a previous time scale position, that time scale position is marked as a business growth position, and the segment between that position and the previous position is recorded as part of a business growth segment. When multiple consecutive time scale positions show a continuous increase in the number of business processing records, these time scale positions are merged consecutively to form a continuous business growth segment. When the number of records at a later time scale position is consistent with that at a previous time scale position, that position is recorded as part of a stable business segment. When the number of records at a later time scale position is lower than that at a previous time scale position, that position is recorded as part of a business decline segment. Through the above sorting process, the sequence of changes in business volume can be divided into multiple continuous time segments according to the time progression, including business growth segments, business stability segments, and business decline segments.

[0040] After segmenting the business volume change sequence, the actual growth segments are extracted from all segments. By reading the time scale positions of each business growth segment, the start and end time scale positions corresponding to each segment are recorded as the time range of that actual growth segment. Within this time range, the number of business processing records is cumulatively counted. During this cumulative counting process, the number of business processing records corresponding to all time scale positions within the actual growth segment is continuously recorded, ensuring that each actual growth segment contains complete time range information and business processing scale change information. After recording all actual growth segments, they are consecutively arranged in chronological order to form a set of actual growth segments. This set of actual growth segments clearly reflects the growth stages of the business processing scale during operation and presents the distribution of these growth stages over time.

[0041] After forming a set of real growth segments, the resource allocation rhythm is adjusted based on these segments as the trigger for capacity expansion. During operation, when time progresses to the starting time marker of a real growth segment, the changes in the business processing scale recorded in that segment are used as the basis for resource allocation adjustment, gradually adjusting the resource allocation quantity according to the changes in the number of business processing records recorded in the segment. As the real growth segment continues to advance, the resource allocation process proceeds continuously according to the order of changes in business processing scale by reading the time markers and corresponding business processing records for each segment. When time progresses to the ending time marker of a real growth segment, the resource allocation quantity remains stable based on the number of business processing records corresponding to the stable business segments recorded in the business volume change sequence, ensuring that the resource investment process maintains a consistent relationship with the business processing scale. Through this resource allocation rhythm adjustment method, the entire operation process is arranged according to the real growth segments, thus forming a stable operation process and avoiding the impact of abnormal business volume changes caused by duplicate records on the resource allocation process.

[0042] This invention constructs a continuous operation sequence by uniformly organizing the continuous operation records generated during the operation of each node in a chronological order and introducing node status markers into the organized records. Based on this continuous operation sequence, parallel segments formed during network outages are identified, and the locations of repetitive business operations are further located. By backtracking and organizing repetitive segments to form a unique trajectory sequence, the multi-node parallel advancement records originally formed during network fluctuations can be replaced by a unified data advancement trajectory. This eliminates duplicate records in the data processing process, allowing the data statistics process to be organized based on the actual data advancement relationship and avoiding abnormal amplification of business volume statistics results due to the superposition of duplicate records.

[0043] This invention reorganizes the business volume change sequence based on a deduplicated unified sequence, extracts the actual growth segments from the business volume change sequence, and uses these actual growth segments as the basis for adjusting the resource allocation rhythm to trigger capacity expansion. This ensures that the resource input process progresses gradually according to the actual business growth trajectory, thereby maintaining consistency between resource scheduling behavior and changes in business processing scale. Through this method, resource allocation behavior during operation can be controlled based on actual business changes, ensuring a stable overall operation and preventing the impact of abnormal business volume changes caused by duplicate recordings on resource scheduling.

[0044] This invention provides, for example Figure 2 The heterogeneous data integration and processing system shown includes a runtime record construction module, a parallel segment recognition module, a trajectory generation module, a unified sequence reconstruction module, and a resource scheduling and control module. The operation record construction module collects the continuous operation records corresponding to the heterogeneous data processed by each node during operation, organizes the continuous operation records according to a unified time order, and writes node status markers into the organized continuous operation records to form a continuous operation sequence. The parallel segment identification module compares the data advancement rhythm of each node based on the continuous running sequence, extracts the parallel segments formed by the independent advancement of each node during the network interruption in the continuous running sequence, marks the positions of repeated business operations in the parallel segments, and generates a list of repeated segments. The trajectory generation module traces back the continuous running sequence based on the list of repeated segments, extracts the original operation trajectory in the corresponding segment of the list of repeated segments, rearranges the original operation trajectory according to the order of business source, and generates a unique trajectory sequence. The unified sequence reconstruction module rearranges the continuous running sequence based on the unique trajectory sequence. It replaces the corresponding segments of the duplicate segment list in the continuous running sequence with the unique trajectory sequence, and generates a deduplicated unified sequence while maintaining the unified time progression order. The resource scheduling and control module reorganizes the business volume change sequence based on the deduplicated unified sequence, extracts the real growth segment from the business volume change sequence, and uses the real growth segment as the basis for expansion triggering to adjust the resource allocation rhythm, forming a stable operation process.

[0045] The heterogeneous data integration and processing method provided in this embodiment of the invention is implemented through the above-mentioned heterogeneous data integration and processing system. For details of the specific method and process of the heterogeneous data integration and processing system, please refer to the above-mentioned embodiment of the heterogeneous data integration and processing method, which will not be repeated here.

[0046] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for integrating and processing heterogeneous data, characterized in that, Includes the following steps: Collect the continuous operation records corresponding to the heterogeneous data processed by each node during operation, organize the continuous operation records according to a unified time order, and write node status flags into the organized continuous operation records to form a continuous operation sequence. Based on the continuous running sequence, the data advancement rhythm of each node is compared. Parallel segments formed by the independent advancement of each node during the network interruption are extracted from the continuous running sequence. The positions of repeated business operations are marked in the parallel segments to generate a list of repeated segments. Based on the list of repeated segments, trace back the continuous operation sequence, extract the original operation trajectory in the corresponding segment of the list of repeated segments, rearrange the original operation trajectory according to the order of business source, and generate a unique trajectory sequence. The continuous running sequence is rearranged based on the unique trajectory sequence. The unique trajectory sequence is used to replace the corresponding segment of the duplicate segment list in the continuous running sequence, and a deduplicated unified sequence is generated while maintaining the unified time progression order. Based on the deduplicated unified sequence, the business volume change sequence is reorganized, and the real growth segment is extracted from the business volume change sequence. The real growth segment is used as the basis for expansion triggering to adjust the resource allocation rhythm, forming a stable operation process.

2. The heterogeneous data integration and processing method according to claim 1, characterized in that, The steps for forming a continuous running sequence are as follows: Collect operation records corresponding to data reading, data parsing, data conversion, data writing, and data forwarding actions generated during the processing of heterogeneous data at each node. Write node identification information, business source identification information, data source identification information, and operation occurrence time information into the operation records to form a continuous operation record set. Read the operation occurrence time information from the continuous operation record set, and organize the operation records in the continuous operation record set according to a unified time scale to form a unified time progression structure; Read the node running status information in the unified time advancement structure, write the node status flags corresponding to the normal operation status, network interruption status and network recovery status into the operation record, and form an operation record containing node status flags; The operation records containing node status markers are arranged continuously according to a unified time progression structure to form a continuous operation sequence that unfolds in the order of time progression.

3. The heterogeneous data integration and processing method according to claim 2, characterized in that, The steps to generate the list of repeating fragments are as follows: Read the node identification information, business source identification information, data source identification information, operation occurrence time information, and node status markers in the continuous running sequence, scan and organize the continuous running sequence according to a unified time progression order, and form a node running trajectory comparison structure on a unified time axis; Based on the node running trajectory comparison structure, the network interruption status information in the node status marker is read, the network interruption section is located on a unified time axis, and the parallel segments formed by the independent advancement of each node in the network interruption section are extracted from the continuous running sequence. Read the business source identification information and operation occurrence time information in the parallel segment, arrange the operation records in the parallel segment according to the business source relationship, and mark the position of duplicate business operation at the position corresponding to the same business source identification information; Read the time segment information corresponding to the parallel segments, the node operation trajectory information, and the location marking information of repeated business operations, and record them centrally in the order of time progression to form a list of repeated segments.

4. The heterogeneous data integration and processing method according to claim 3, characterized in that, The list of repeated segments records time segment information, node operation trajectory information, and location marking information of repeated business operations. These are arranged continuously in chronological order, maintaining the correspondence between node operation trajectory and business source identification information within each time segment, thus forming a data record structure.

5. The heterogeneous data integration and processing method according to claim 3, characterized in that, The steps for generating a unique trajectory sequence are as follows: Based on the time segment information in the list of repeated segments, the operation records within the corresponding segments are extracted from the continuous running sequence to form the original operation trajectory set; Read the business source identification information from the original operation trajectory set, merge and organize the operation records in the original operation trajectory set according to the business source identification information, and keep the arrangement relationship between the operation occurrence time information and the node identification information unchanged to form a business source advancement trajectory; The operation records are arranged sequentially based on the business source identification information in the business source advancement trajectory, and the operation occurrence time sequence is continuously unfolded within the business source advancement trajectory to form a unified business advancement sequence; The operation records are arranged continuously according to the unified business progress order, and a unique trajectory sequence is generated while maintaining the consistency of node identification information, business source identification information and operation occurrence time information.

6. The heterogeneous data integration and processing method according to claim 5, characterized in that, The operation records in the unified business advancement sequence are arranged continuously according to the business source identification information. The operation records corresponding to the same business source identification information form a continuous advancement relationship according to the operation occurrence time information. The operation records corresponding to different business source identification information are arranged according to the order of the business source identification information, forming a unified business advancement structure in a unique trajectory sequence.

7. The heterogeneous data integration and processing method according to claim 5, characterized in that, The steps for generating the deduplicated unified sequence are as follows: Based on the time segment information in the list of repeated segments, locate the corresponding segment in the continuous running sequence, extract the operation record, and form a segment operation record set; Read the business source identifier information, operation occurrence time information, and node identifier information from the unique trajectory sequence, expand and arrange the operation records in the unique trajectory sequence according to the order of business source, and map them to the corresponding set of segment operation records; Replace the set of segment operation records for the corresponding time segment in the continuous running sequence with the operation records in the unique trajectory sequence, and insert the operation records from the unique trajectory sequence into the corresponding time segment; Read the operation time information in the continuous running sequence, arrange the continuous running sequence after the replacement is completed in a unified time progression order, and generate a unified sequence after deduplication.

8. The heterogeneous data integration and processing method according to claim 7, characterized in that, The steps for generating the business volume change sequence and adjusting the resource allocation rhythm are as follows: Read the operation records in the deduplicated unified sequence, establish a unified time scale statistical structure based on the operation occurrence time information, count the number of business processing records at the corresponding time scale position in the unified time scale statistical structure, and form a business volume change sequence. Read the number of business processing records in the business volume change sequence, compare and organize the number of business processing records at adjacent time scale positions, and divide the business volume change sequence into business growth segment, business stability segment, and business decline segment. Read the time scale position and the number of business processing records corresponding to the business growth segment, extract the records of the business growth segment, and form a set of real growth segments; Based on the time scale position and the number of business processing records in the set of actual growth segments, the resource allocation rhythm is adjusted to form a stable operation process under the condition of unified time advancement sequence.

9. A heterogeneous data integration and processing system, used to implement the heterogeneous data integration and processing method according to any one of claims 1-8, characterized in that, It includes a runtime record construction module, a parallel segment recognition module, a trajectory generation module, a unified sequence reconstruction module, and a resource scheduling and control module. The operation record construction module collects the continuous operation records corresponding to the heterogeneous data processed by each node during operation, organizes the continuous operation records according to a unified time order, and writes node status markers into the organized continuous operation records to form a continuous operation sequence. The parallel segment identification module compares the data advancement rhythm of each node based on the continuous running sequence, extracts the parallel segments formed by the independent advancement of each node during the network interruption in the continuous running sequence, marks the positions of repeated business operations in the parallel segments, and generates a list of repeated segments. The trajectory generation module traces back the continuous running sequence based on the list of repeated segments, extracts the original operation trajectory in the corresponding segment of the list of repeated segments, rearranges the original operation trajectory according to the order of business source, and generates a unique trajectory sequence. The unified sequence reconstruction module rearranges the continuous running sequence based on the unique trajectory sequence. It replaces the corresponding segments of the duplicate segment list in the continuous running sequence with the unique trajectory sequence, and generates a deduplicated unified sequence while maintaining the unified time progression order. The resource scheduling and control module reorganizes the business volume change sequence based on the deduplicated unified sequence, extracts the real growth segment from the business volume change sequence, and uses the real growth segment as the basis for expansion triggering to adjust the resource allocation rhythm, forming a stable operation process.

Images