A system for building a medical business data lake warehouse based on a distributed architecture

By building a medical business data lake warehouse system based on a distributed architecture, and splitting data tables into synchronous micro-batches for staggered writing, the problems of lock waiting and transaction conflicts in multi-threaded concurrent write mode were solved, improving data synchronization throughput and ensuring data integrity.

CN123115281APending Publication Date: 2026-11-17JIANGSU BORONG INFORMATION TECHNOLOGY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611363929.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-04
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing data synchronization solutions often employ conventional multi-threaded concurrent write modes when delivering data to distributed databases. This can easily lead to lock waits or transaction conflicts between concurrent write tasks, increasing the system load on database nodes and limiting the overall throughput of the data synchronization chain.

Method used

The system is built using a distributed architecture-based medical business data lake warehouse. The data acquisition module forms a set of data table features, the modeling module generates business relationship graphs and resource conflict graphs, the clustering module splits the data table into synchronous micro-batches, the synchronization module writes the data, and the synchronization parameters are adjusted through the judgment module and the optimization module to optimize the data synchronization process.

Benefits of technology

It effectively alleviates the pressure of lock waiting and transaction retry in high-concurrency scenarios of distributed databases, improves the overall data synchronization throughput, and avoids downstream analysis platforms reading incomplete dirty data. The system can smoothly adapt to sudden business traffic and reduce recalculation overhead.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115281A_ABST
    Figure CN123115281A_ABST
Patent Text Reader

Abstract

This invention relates to the field of database integration technology, and more particularly to a medical business data lake warehouse construction system based on a distributed architecture. The system includes: a data acquisition module, which collects business identifiers, query records, arrival times, data synchronization component status, and distributed data lake warehouse operating status of data tables to form a data table feature set; and a modeling module, which determines the association reliability based on the business identifier matching rate, query co-occurrence frequency, and arrival time difference of the data table feature set, and determines the resource conflict degree based on the synchronization channel time overlap rate and the distributed data lake warehouse transaction retry ratio, generating a business association graph and a resource conflict graph. In this invention, data tables with high resource conflict degrees are split into different synchronization micro-batches for staggered writing, effectively alleviating the lock waiting and transaction retry pressure of the distributed database in high-concurrency scenarios, and improving the overall data synchronization throughput while ensuring the data flow order.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of database integration technology, and in particular to a system for building a medical business data lake warehouse based on a distributed architecture. Background Technology

[0002] With the continuous development of medical informatization, hospitals have deployed numerous independent and heterogeneous business systems, generating massive amounts of medical data with complex logical relationships. In order to break down data silos and support unified operational analysis and clinical decision support, medical institutions usually need to build a global operational data storage center. In recent years, distributed databases have gradually become the preferred carrier for underlying data aggregation due to their good horizontal scalability and high availability. In practical applications, it is usually necessary to use data synchronization components to extract and write business change logs from various source production databases to the distributed storage center in real time and with high concurrency.

[0003] Existing data synchronization solutions often employ conventional multi-threaded concurrent write modes when delivering data to distributed databases. However, when processing large volumes of business-related data, they lack consideration for the underlying transaction mechanisms and resource contention on the target side. This can easily lead to lock waits or transaction conflicts between concurrent write tasks, resulting in numerous transaction rollbacks and retries. This not only increases the system load on database nodes but also limits the throughput of the overall data synchronization chain. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a medical business data lake warehouse construction system based on a distributed architecture, which aims to improve the problem that existing data synchronization solutions often use conventional multi-threaded concurrent write modes when delivering data to distributed databases.

[0005] This invention provides the following technical solution: a medical business data lake warehouse construction system based on a distributed architecture, comprising: The data acquisition module collects the business identifiers, query records, arrival times, data synchronization component status, and distributed data lake warehouse operation status of the data tables, forming a set of data table features. The modeling module determines the association reliability based on the business identifier matching rate, query co-occurrence frequency and arrival time difference of the feature set of the data table, and determines the resource conflict degree based on the synchronization channel time overlap rate and the distributed data lake warehouse transaction retry ratio, and generates a business association graph and a resource conflict graph. The clustering module clusters the data tables whose association credibility reaches a first preset threshold into association clusters based on the business association graph, and splits the association clusters into synchronous micro-batches based on the resource conflict graph, generating synchronization parameters including synchronization order, concurrency and batch data volume. The synchronization module writes the data corresponding to the micro-batch into the distributed data lake warehouse according to the synchronization parameters, forming the source endpoint, synchronization task progress, and data storage progress. The determination module determines the completeness of the associated cluster based on the source endpoint, the synchronization task progress, the database entry progress, and the data mismatch ratio. When the completeness reaches a second preset threshold, a queryable identifier is formed. The optimization module adjusts the synchronization parameters based on the retry ratio of the distributed data lake warehouse transactions, the backlog represented by the synchronization task progress, and the integrity. When the deviation between the association reliability and the resource conflict degree reaches a third preset threshold for a consecutive preset number of periods, the association cluster is locally clustered, the synchronization parameters are updated, and the results are returned to the synchronization module.

[0006] Preferably, in the acquisition module, forming the feature set of the data table specifically includes the following steps: Based on the global timestamp synchronization service, the collected business identifier, query record and arrival time sequence are aligned on the time axis to construct a business feature vector; Capture the memory queue status of the data synchronization component and the concurrent lock waiting status of the distributed data lake warehouse to construct a physical feature vector; The business feature vector and the physical feature vector are concatenated into a matrix to form a data table feature set that includes business logic dimension and underlying resource dimension.

[0007] Preferably, in the modeling module, generating the business relationship diagram and resource conflict diagram specifically includes the following steps: Using the data table as a graph node, and taking the positive weighted value of the business identifier matching rate and the query co-occurrence frequency as the base, and combining it with the exponential decay function of the arrival time difference for penalty, the edge weight is calculated to generate a business association graph. Using the synchronization channel time overlap rate as the base, and combining it with the nonlinear amplification coefficient of the transaction retry ratio of the distributed data lake warehouse, the probability of concurrent writes causing lock conflicts is calculated as the edge weight to generate a resource conflict graph.

[0008] Preferably, the step of penalizing the edge weights by combining the exponential decay function of the arrival time difference with the calculation of edge weights to generate the business association graph specifically includes the following steps: Extract the business identifier matching rate and the query co-occurrence frequency, normalize them, and dynamically weight them to obtain an initial baseline association value; Identify the medical business scenario category that generated the data table, and match the corresponding adaptive time decay coefficient according to the scenario category, wherein the decay coefficient of the first scenario is greater than the decay coefficient of the second scenario. The adaptive time decay coefficient and the time difference of arrival are substituted into the natural exponential decay formula to generate a dynamic penalty term; Multiply the initial baseline association value by the dynamic penalty term, and output the decayed value as the edge weight of the business association graph.

[0009] Preferably, in the clustering module, the step of splitting the associated clusters into synchronous micro-batches and generating synchronization parameters including synchronization order, concurrency, and batch data volume specifically includes the following steps: Based on the strong dependency directed edges between nodes in the business association graph, the data tables within the association cluster are topologically sorted to determine the synchronization order; Using the inverse of the resource conflict degree in the resource conflict graph as the constraint knapsack capacity, the sorted data table sequence is dynamically programmed and divided into multiple synchronous micro-batches. Based on the graph connectivity density of the nodes within a single micro-batch, the batch data volume is allocated in a positive direction and the concurrency is limited in a negative direction to generate synchronization parameters.

[0010] Preferably, the step of dynamically programming and dividing the sorted data table sequence into multiple synchronous micro-batches specifically includes the following steps: Initialize the dynamic programming state matrix to record the target batch partitioning cost of the data table sequence at different cut points; For any candidate interval in the data table sequence, extract the peak edge weights of all data tables in the interval in the resource conflict graph, and determine the upper limit of the interval capacity of the candidate interval by the reciprocal of the peak edge weights; Calculate the cumulative data volume within the candidate interval. If the cumulative data volume exceeds the upper limit of the interval capacity, mark the candidate interval as unreachable in the state matrix. Based on the state transition equation, and under the premise of avoiding the unreachable state, the set of target cutting points that makes the number of global batch divisions meet the preset cost condition is solved, and synchronous micro-batches are output according to the set of target cutting points.

[0011] Preferably, in the synchronization module, the steps of forming the source endpoint, the synchronization task progress, and the database entry progress specifically include the following steps: Parse the redo log sequence number of the source database and record it as the source location; Extract the buffer queue consumption offset between the read thread and the write thread inside the data synchronization component and record it as the synchronization task progress; Obtain the global transaction timestamp of the distributed data lake warehouse cluster, extract the high watermark of committed transactions, and construct a status tracking link to form the data storage progress.

[0012] Preferably, in the determination module, determining the completeness of the associated cluster specifically includes the following steps: Calculate the relative displacement difference between the source endpoint, the synchronization task progress, and the data drop progress, and map it to the data physical readiness rate; Extract the proportion of data in the micro-batch that is not matched due to missing foreign key dependencies, and map it to the business logic isolation rate; The completeness of the associated cluster is calculated by dynamically weighting and fusing the physical readiness rate and the reciprocal of the logical isolation rate.

[0013] Preferably, in the optimization module, adjusting the synchronization parameters and performing local clustering of the associated clusters specifically includes the following steps: Calculate the joint gradient between the retry ratio of the distributed data lake warehouse transactions and the backlog, and dynamically increase or decrease the concurrency and batch data volume of subsequent micro-batches through a feedback controller; When a deviation that reaches the third preset threshold is identified, the data table that caused the deviation is locked, and a local subgraph within the associated cluster is extracted with the data table as the center. The overall topology of the non-subgraph portion within the original associated cluster remains unchanged. Only the local subgraph is re-performed with constraint clustering to generate updated synchronization parameters, which are then returned to the synchronization module.

[0014] The present invention has the following beneficial effects: 1. In this invention, data tables with high resource conflict are split into different synchronous micro-batches for staggered writing, which effectively alleviates the pressure of lock waiting and transaction retry in high-concurrency scenarios of distributed databases, and improves the overall data synchronization throughput while ensuring the data flow order.

[0015] 2. In this invention, the timing of opening synchronized data is determined by comprehensively evaluating the progress of the underlying physical flow and the foreign key matching status of the upper-level medical records. This effectively intercepts incomplete data in the intermediate gap state and avoids the downstream analysis platform reading incomplete dirty data.

[0016] 3. In this invention, the concurrency parameters of subsequent batches are dynamically adjusted based on the real-time synchronization backlog and transaction error frequency, and lightweight reconstruction is performed only on the affected area when the local data association relationship decays, so that the system can smoothly adapt to sudden business traffic and reduce recalculation overhead. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method for constructing a medical business data lake warehouse based on a distributed architecture, as proposed in this invention. Figure 2This is a flowchart of the micro-batch splitting and synchronization parameter generation process of the clustering module of this invention; Figure 3 This is a flowchart of the process for determining the integrity of associated clusters and generating queryable identifiers in the determination module of this invention. Detailed Implementation

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This invention provides a system for building a medical business data lake warehouse based on a distributed architecture, such as... Figure 1 As shown, it includes: The data acquisition module collects the business identifiers, query records, arrival times, data synchronization component status, and distributed data lake warehouse operation status of the data tables, forming a set of data table features. Furthermore, in the data acquisition module, forming the feature set of the data table specifically includes the following steps: Based on the global timestamp synchronization service, the collected business identifiers, query records and arrival times are aligned with the timeline to construct a business feature vector. Capture the memory queue state of the data synchronization component and the concurrent lock waiting state of the distributed data lake warehouse to construct a physical feature vector; The business feature vector and the physical feature vector are concatenated into a matrix to form a data table feature set that includes both business logic dimension and underlying resource dimension.

[0020] Specifically, the acquisition module obtains the business identifiers, query records, and arrival times of data tables from the medical production system, and also collects the status of data synchronization components and the operational status of the distributed data lake warehouse. The system records the acquisition time of each data item through a global timestamp synchronization service, converts the source server time to a unified time base, and aligns the data from different systems along the timeline according to a 5-minute acquisition window. For any two data tables, the system extracts the business identifier matching rate, query co-occurrence frequency, and arrival time difference to construct a business feature vector, the expression of which is: ; in, Represents a data table With data table Business feature vectors between This indicates the matching rate of business identifiers between the two data tables based on the patient's unique identification code or medical record number. This indicates the frequency at which two data tables appear together in the same query. This represents the average absolute time difference in arrival for corresponding records from two data tables entering the data synchronization component, in hours.

[0021] The system simultaneously collects the memory buffer queue usage ratio through the monitoring interface of the data synchronization component, and collects the lock wait time corresponding to the synchronization transaction through the distributed data lake warehouse operation index interface, constructing a physical feature vector, the expression of which is: ; in, Represents a data table With data table The physical feature vector corresponding to the synchronous task. This indicates the percentage of the buffer queue occupied by the corresponding synchronization task. This represents the average lock wait time for the corresponding synchronous transaction within the acquisition window. The system performs range normalization on features of different dimensions, with the following formula: ; in, This represents the eigenvalues ​​to be normalized. and These represent the minimum and maximum values ​​of the same feature within the current acquisition window, respectively. At that time, the system will set the normalization result to 0.

[0022] The system follows the formula The splicing of business characteristics and physical characteristics, among which, This represents the combined feature vector corresponding to the two data tables. This represents the vector concatenation operator. and These represent the business feature weights and physical feature weights, respectively. This embodiment will... Set to 0.7, Set to 0.3. The comprehensive feature vectors corresponding to all data tables constitute the feature set of the data tables.

[0023] This unified the time benchmark for business data and operational status, providing input data for subsequent correlation modeling.

[0024] The modeling module determines the association reliability based on the business identifier matching rate of the data table feature set, the query co-occurrence frequency, and the arrival time difference. It also determines the resource conflict degree based on the synchronization channel time overlap rate and the distributed data lake warehouse transaction retry ratio, and generates a business association graph and a resource conflict graph. Furthermore, the modeling module generates business relationship diagrams and resource conflict diagrams, specifically including the following steps: The data table is used as a graph node. The positive weighted value of the business identifier matching rate and query co-occurrence frequency is used as the base. The edge weight is calculated by combining the exponential decay function of arrival time difference for penalty and generating a business association graph. Using the synchronization channel time overlap rate as the base, and combining the non-linear amplification coefficient of the transaction retry ratio in the distributed data lake warehouse, the probability of concurrent writes causing lock conflicts is calculated as the edge weight to generate a resource conflict graph.

[0025] Furthermore, by incorporating an exponential decay function of arrival time difference as a penalty, the calculation of edge weights to generate a business association graph specifically includes the following steps: Extract the business identifier matching rate and query co-occurrence frequency, normalize and dynamically weight the calculation to obtain the initial baseline association value; Identify the medical business scenario category that generates the data table, and match the corresponding adaptive time decay coefficient according to the scenario category, wherein the decay coefficient of the first scenario is greater than the decay coefficient of the second scenario. The adaptive time decay coefficient and the time difference of arrival are substituted into the natural exponential decay formula to generate a dynamic penalty term; Multiply the initial baseline association value by the dynamic penalty term, and output the decayed value as the edge weight of the business association graph.

[0026] Specifically, based on the feature set of the data tables formed by the acquisition module, the modeling module uses the data tables as graph nodes to calculate the association reliability and resource conflict degree between any two data table nodes, generating a business association graph and a resource conflict graph respectively. The system calculates the business identifier matching rate based on the business identifier sets of the two data tables, using the following formula: ; in, and Representing data tables With data table The collection of the patient's unique identification code or medical record number within the current data collection window. Indicates the number of shared service identifiers. This represents the total number of unique business identifiers after deduplication. When the total number of unique business identifiers after deduplication is 0, the system will assign a security value of 0 to the matching rate of that business identifier.

[0027] The system parses the table names and join relationships in the query logs and calculates the query co-occurrence frequency using the following formula: ; in, This indicates that data tables are accessed simultaneously within the current data collection window. With data table The number of queries, and These represent accessing data tables. and data table The system assigns a co-occurrence frequency of the query to 0 when the total number of queries in the denominator is 0.

[0028] The system follows the formula Calculate the initial baseline correlation value, where, This represents the initial baseline association value between two data table nodes. and This represents the positive weighting coefficient. In this embodiment, Set to 0.6, Set it to 0.4.

[0029] The system determines the medical business scenario category based on the business system tags in the data table metadata and configures a time decay coefficient for different scenarios. The first scenario includes outpatient registration and prescription data, with a time decay coefficient set to 0.8 per hour. The second scenario includes inpatient medical orders and inpatient settlement data, with a time decay coefficient set to 0.2 per hour. The system calculates dynamic penalty items: ; in, Represents a data table With data table Dynamic penalty items, Represents the natural constant. This represents the time decay coefficient determined based on the scenarios represented by the two data tables. It represents the average absolute time difference of arrival, measured in hours.

[0030] The system follows the formula Calculate the association confidence level, where, Data tables in the business relationship diagram With data table The system writes the edge weights between data table nodes and their corresponding edge weights into the adjacency matrix to generate a business association graph.

[0031] When generating the resource conflict graph, the system determines the dwell time interval of the corresponding tasks in the two data tables based on the collected entry and exit times of synchronization tasks in the buffer queue, and calculates the time overlap rate of the synchronization channel: ; in, and Representing data tables With data table The corresponding dwell time interval for synchronous tasks. Indicates the duration of overlap. This represents the total duration after merging. When the total duration after merging is 0, the system will safely assign a time overlap rate of 0 to the synchronization channel.

[0032] The data synchronization component records the number of transaction commits and retries according to the target data table and micro-batch identifier, and then calculates the retry ratio of distributed data lake warehouse transactions involving two data tables: ; in, This indicates the number of retries for transactions involving two data tables within the current data collection window. This indicates the number of successful commits. When the total number of commits and retries for all transactions involving these two tables is 0, the system will safely assign a retry ratio of 0 to the distributed data lake warehouse transaction.

[0033] System computing resource conflict degree ,in, Data table in resource conflict graph With data table The degree of resource conflict between them This represents the nonlinear amplification factor and is set to 2.0. The system writes the resource conflict degree into the adjacency matrix and generates a resource conflict graph.

[0034] This transforms inter-table business relationships and synchronous resource competition into graph edge weights, providing a basis for the partitioning of associated clusters and micro-batch scheduling.

[0035] like Figure 2 As shown, the clustering module clusters data tables whose association credibility reaches the first preset threshold into association clusters based on the business association graph, and splits the association clusters into synchronous micro-batches based on the resource conflict graph, generating synchronization parameters including synchronization order, concurrency and batch data volume. Furthermore, in the clustering module, the associated clusters are split into synchronous micro-batches, and synchronization parameters including synchronization order, concurrency, and batch data volume are generated. Specifically, this includes the following steps: Based on the strong dependency directed edges between nodes in the business association graph, the data tables within the association cluster are topologically sorted to determine the synchronization order; Using the inverse of the resource conflict degree in the resource conflict graph as a constraint on the knapsack capacity, the sorted data table sequence is dynamically programmed and divided into multiple synchronous micro-batches. Based on the graph connectivity density of nodes within a single micro-batch, the batch data volume is allocated in the forward direction and the concurrency is limited in the reverse direction to generate synchronization parameters.

[0036] Furthermore, the sorted data table sequence is dynamically divided into multiple synchronous micro-batches using dynamic programming, specifically including the following steps: Initialize the dynamic programming state matrix to record the target batch partitioning cost of the data table sequence at different cut points; For any candidate interval in the data table sequence, extract the peak edge weights of all data tables in the interval in the resource conflict graph, and determine the upper limit of the interval capacity of the candidate interval by the reciprocal of the peak edge weights. Calculate the cumulative data volume within the candidate interval. If the cumulative data volume exceeds the interval capacity limit, mark the candidate interval as unreachable in the state matrix. Based on the state transition equation, and under the premise of avoiding unreachable states, the set of target cutting points that makes the number of global batch divisions meet the preset cost condition is solved, and synchronous micro-batches are output according to the set of target cutting points.

[0037] Specifically, based on the business association graph and resource conflict graph, the clustering module connects data table nodes whose edge weights in the business association graph reach a first preset threshold into the same association cluster. In this embodiment, the first preset threshold is set to 0.5, and each association cluster is obtained through connected component search. The system determines the data write dependency direction between data tables according to the primary and foreign key mapping in the business data dictionary. For data tables lacking primary and foreign key relationships, the system determines the dependency direction according to the generation time of the business records. The system generates strongly dependent directed edges in the manner of primary table pointing to secondary table, and obtains the synchronization order through Kahn topological sorting. When a circular dependency occurs, the system removes the non-primary and foreign key edges with the lowest association confidence value within the ring, and then performs topological sorting. The system performs dynamic planning to cut the sorted data table sequence. For the starting position is And the ending position is For candidate intervals, extract the peak edge weights of data table nodes within the interval in the resource conflict graph, and calculate the upper limit of the interval capacity: ; in, This represents the upper limit of the capacity of the candidate interval. This represents the capacity conversion constant and is set to 50000 rows. This represents the peak value of the resource conflict edge weights within the candidate interval. This represents a smoothing constant to prevent the denominator from being zero and is set to 0.01. The cumulative data volume of the candidate interval is calculated according to the formula... Calculate, where, This indicates the cumulative number of rows of data to be synchronized in the candidate interval. Indicates the first position in the sorted sequence The number of rows of data to be synchronized in each data table. When When the time comes, mark the candidate interval as a reachable state. Otherwise, mark it as unreachable. .

[0038] System Settings and according to the formula Calculate the batch partitioning cost at different cutting points, where, Indicates the preceding The cost of dividing the target batches for each data table. Indicates the candidate cutting points. The constraints of the operation are: and The system records the cutting points that enable the state transition, and obtains the target cutting point set through backtracking, thereby outputting synchronous micro-batches.

[0039] For each synchronized micro-batch, the system calculates its directed graph connectivity density. ,in, This represents the directed graph connectivity density of the data table nodes within a micro-batch. Indicates the total number of nodes. This represents the number of strongly dependent directed edges between nodes. When At that time, Set to 0.

[0040] The system sets the base batch size to 2000 rows, the base concurrency to 16, and calculates the batch data volume. Concurrency ,in, Indicates the batch data volume. Indicates the basic batch quantity. Indicates the degree of concurrency. This represents the base concurrency level, and the floor function is used to generate integer parameters. The system combines the synchronization order, concurrency level, and batch data size into synchronization parameters.

[0041] This preserves the order of data table dependencies while assigning data tables with high resource conflict to different synchronization micro-batches.

[0042] The synchronization module writes the data corresponding to the micro-batch into the distributed data lake warehouse according to the synchronization parameters, forming the source endpoint, synchronization task progress, and data storage progress. Furthermore, the synchronization module, which establishes the source endpoint, synchronization task progress, and database download progress, specifically includes the following steps: Parse the redo log sequence number of the source database and record it as the source location; Extract the consumption offset of the buffer queue between the read thread and the write thread inside the data synchronization component and record it as the progress of the synchronization task. Obtain the global transaction timestamps of the distributed data lake warehouse cluster, extract the high watermarks of committed transactions, and construct a status tracking link to form the data storage progress.

[0043] Specifically, the synchronization module receives synchronization parameters generated by the clustering module, reads the change logs of the source database according to the synchronization order and batch data volume, and writes the micro-batch data to the distributed data lake warehouse according to the concurrency. The system parses the sequence position identifiers in the redo logs of the source database and converts them into comparable sequence positions according to the log order of the source database.

[0044] For including Synchronization micro-batches of log events, with the source endpoint set according to the formula It is confirmed that, among them, This indicates the source endpoint of the current synchronized micro-batch. Indicates the first The redo log sequence position of a change log event. This indicates the total number of change log events contained in the micro-batch. The data synchronization component assigns a sequential sequence number to each micro-batch and writes the data generated by the read thread to the partition buffer queue.

[0045] For a task containing multiple buffer queue partitions, the system extracts the offsets of each partition that have been consumed consecutively, and records the minimum value among them as the synchronization task progress. ,in, Indicates the progress of the synchronization task. Indicates the first The consecutive consumption offsets of each buffer queue partition. This indicates the total number of partitions in the buffer queue. Using the minimum consumption offset avoids classifying data that has not yet been processed by some partitions as consumed. The synchronization component obtains the global transaction timestamp for each write transaction through the distributed data lake warehouse transaction interface and records the micro-batch sequence number, transaction commit result, and commit timestamp.

[0046] The system checks the submission status according to the micro-batch sequence number and determines the submission level by the last sequence number of consecutive submissions starting from the start sequence number. ,in, This indicates the water level sequence number of the micro-batch submitted consecutively. The largest candidate number to participate in the water level calculation. Indicates the sequence number is The micro-batch submission status is set to 1 upon successful submission, and 1 when the set is empty (i.e., the starting sequence number). (The micro-batch has not yet been successfully submitted), the system will submit the water level sequence number. Security is set to 0.

[0047] The system records the commit timestamp of the transaction corresponding to the water level sequence number as the progress of the water level settling process. ,in, Indicates the progress of data storage in the distributed data lake warehouse. This indicates the global transaction timestamp when the transaction corresponding to the water level sequence number was completed and committed. The system also maintains checkpoint records, which at least include the time window identifier, associated cluster identifier, micro-batch sequence number, source endpoint, number of consumption records, number of commit records, and commit timestamp.

[0048] This establishes a progress tracking link between the source end, the synchronization channel, and the target end of the distributed data lake warehouse.

[0049] like Figure 3 As shown, the determination module determines the completeness of the associated cluster based on the source endpoint, synchronization task progress, data storage progress, and data mismatch ratio. When the completeness reaches the second preset threshold, a queryable identifier is formed. Furthermore, in the determination module, determining the completeness of the associated cluster specifically includes the following steps: Calculate the relative displacement difference between the source endpoint, the synchronization task progress, and the data drop progress, and map it to the data physical readiness rate. Extract the proportion of data in the micro-batch that is not matched due to missing foreign key dependencies, and map it to the business logic isolation rate; The completeness of the associated cluster is calculated by dynamically weighting and fusing the reciprocals of the physical readiness rate and the logical isolation rate.

[0050] Specifically, the judgment module reads the checkpoint records maintained by the synchronization module, and summarizes the source-end extracted rows, channel-end consumed rows, and target-end committed rows for all synchronization micro-batches according to the associated cluster and time window, denoted as follows: , and The system calculates the relative displacement difference between the source end and the synchronization channel, and between the synchronization channel and the distributed data lake warehouse.

[0051] The formulas are respectively and ,in, This represents the relative displacement difference between the source and the synchronization channel. This represents the relative displacement difference between the synchronization channel and the distributed data lake warehouse. The system calculates the overall relative displacement difference. and physical readiness rate ,in, Indicates the overall relative displacement difference. This indicates the data physical readiness rate of the associated cluster within the current time window.

[0052] Based on the primary and foreign key mappings in the business data dictionary, the system performs reverse join queries between the child and parent tables in the distributed data lake warehouse, counts the number of data rows in the current association cluster and time window that do not have a parent table related record, and calculates the business logic orphanage rate. ,in, Indicates the business logic isolation rate. This indicates the number of rows for which no related records from the parent table were found. This indicates the total number of rows of data that have been stored in the database within the current associated cluster during the specified time window.

[0053] The system weights and combines the physical readiness rate and the normalized inverse of the business logic orphanage rate using the following formula: ,in, Indicates the completeness of the associated cluster. and These represent physical weights and logical weights, respectively. This represents the smoothing constant and is set to 0.01. This embodiment will... Set it to 0.3, Set to 0.7. When the expected data volume of the time window is 0, the system marks the corresponding window as an empty window based on the source checkpoint. When the expected data volume is greater than 0 and When the second preset threshold of 0.98 is reached, the system writes a queryable identifier for the associated cluster and time window.

[0054] By controlling the data query range based on the physical synchronization status and the matching status of business records, incomplete data is reduced from being read downstream.

[0055] The optimization module adjusts the synchronization parameters based on the retry ratio of distributed data lake warehouse transactions, the backlog and integrity of synchronization task progress. When any deviation between the association credibility and the resource conflict degree reaches the third preset threshold for a consecutive preset number of cycles, the association cluster is locally clustered, the synchronization parameters are updated, and the result is returned to the synchronization module.

[0056] Furthermore, the optimization module adjusts the synchronization parameters and performs local clustering of associated clusters, specifically including the following steps: Calculate the joint gradient of the transaction retry ratio and backlog in the distributed data lake warehouse, and dynamically increase or decrease the concurrency and batch data volume of subsequent micro-batches through the feedback controller; When a deviation that reaches the third preset threshold is detected, the data table that caused the deviation is locked, and a local subgraph within the associated cluster is extracted with the data table as the center. The overall topology of the non-subgraph portion within the original associated cluster remains unchanged. Only constrained clustering is re-executed on the local subgraphs to generate updated synchronization parameters, which are then returned to the synchronization module.

[0057] Specifically, the optimization module calculates the synchronization backlog of the current associated clusters and time windows based on checkpoint records. ,in, Indicates the first The backlog for each monitoring cycle and These represent the number of rows extracted from the source and the number of rows consumed by the channel in the current period, respectively. In this embodiment, the monitoring period is set to 1 minute. The system calculates the periodic difference between the distributed data lake warehouse transaction retry ratio and the backlog. as well as ,in, and These represent the proportion of distributed data lake warehouse transactions in the current period and the previous period, respectively. This represents the difference in the retry ratio. This indicates the difference in backlog.

[0058] The system weights the two time differences to form a joint gradient index. ,in, This represents the joint gradient index for the current period. and These represent the retry penalty coefficient and the backlog penalty coefficient, respectively, and are set to 100 and 0.01. The system is configured with a maximum concurrency of 32, a minimum batch size of 500 rows, a maximum batch size of 10,000 rows, and adjusted synchronization parameters. The concurrency formula is as follows: The formula for batch data volume is: ; in, and These represent the updated concurrency and batch data size, respectively. and These represent the current concurrency level and the batch data volume, respectively. This indicates the completeness of the associated clusters output by the determination module. , and Set them to 1, 500, and 1000 respectively. This means that the calculation results will be restricted to the given upper and lower limits.

[0059] The system recalculates the edge weights of the business association graph and resource conflict graph in each monitoring cycle, and calculates the edge weight deviation. as well as ,in, and They represent the first The correlation reliability deviation and resource conflict deviation between two data tables in a given period are used to determine the baseline period, with the superscript 0 indicating the baseline period. In this embodiment, the third preset threshold is set to 0.3, and the preset number of periods is set to 5. When any deviation reaches 0.3 for five consecutive periods, The system extracts the two data tables connected by the deviation edge and their adjacent data tables of 1 degree as a local subgraph. The system retains the topology and synchronization parameters of the remaining nodes in the associated cluster, only re-performs topology sorting and dynamic programming cutting on the local subgraph, and then returns the updated synchronization order, concurrency degree, and batch data volume to the synchronization module.

[0060] The synchronization parameters are adjusted based on the running status, and the associated clusters are locally updated when the graph relationships continue to change.

[0061] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A system for constructing a medical business data lake warehouse based on a distributed architecture, characterized in that, include: The data acquisition module collects the business identifiers, query records, arrival times, data synchronization component status, and distributed data lake warehouse operation status of the data tables, forming a set of data table features. The modeling module determines the association reliability based on the business identifier matching rate, query co-occurrence frequency and arrival time difference of the feature set of the data table, and determines the resource conflict degree based on the synchronization channel time overlap rate and the distributed data lake warehouse transaction retry ratio, and generates a business association graph and a resource conflict graph. The clustering module clusters the data tables whose association credibility reaches a first preset threshold into association clusters based on the business association graph, and splits the association clusters into synchronous micro-batches based on the resource conflict graph, generating synchronization parameters including synchronization order, concurrency and batch data volume. The synchronization module writes the data corresponding to the micro-batch into the distributed data lake warehouse according to the synchronization parameters, forming the source endpoint, synchronization task progress, and data storage progress. The determination module determines the completeness of the associated cluster based on the source endpoint, the synchronization task progress, the database entry progress, and the data mismatch ratio. When the completeness reaches a second preset threshold, a queryable identifier is formed. The optimization module adjusts the synchronization parameters based on the retry ratio of the distributed data lake warehouse transactions, the backlog represented by the synchronization task progress, and the integrity. When the deviation between the association reliability and the resource conflict degree reaches a third preset threshold for a consecutive preset number of periods, the association cluster is locally clustered, the synchronization parameters are updated, and the results are returned to the synchronization module.

2. The medical business data lake warehouse construction system based on a distributed architecture according to claim 1, characterized in that, In the data acquisition module, the process of forming the feature set of the data table specifically includes the following steps: Based on the global timestamp synchronization service, the collected business identifier, query record and arrival time sequence are aligned on the time axis to construct a business feature vector; Capture the memory queue status of the data synchronization component and the concurrent lock waiting status of the distributed data lake warehouse to construct a physical feature vector; The business feature vector and the physical feature vector are concatenated into a matrix to form a data table feature set that includes business logic dimension and underlying resource dimension.

3. The medical business data lake warehouse construction system based on a distributed architecture according to claim 1, characterized in that, In the modeling module, the generation of business relationship diagrams and resource conflict diagrams specifically includes the following steps: Using the data table as a graph node, and taking the positive weighted value of the business identifier matching rate and the query co-occurrence frequency as the base, and combining it with the exponential decay function of the arrival time difference for penalty, the edge weight is calculated to generate a business association graph. Using the synchronization channel time overlap rate as the base, and combining it with the nonlinear amplification coefficient of the transaction retry ratio of the distributed data lake warehouse, the probability of concurrent writes causing lock conflicts is calculated as the edge weight to generate a resource conflict graph.

4. A medical business data lake warehouse construction system based on a distributed architecture as described in claim 3, characterized in that, The step of penalizing the edge weights by incorporating the exponential decay function of the arrival time difference and generating the business association graph specifically includes the following steps: Extract the business identifier matching rate and the query co-occurrence frequency, normalize them, and dynamically weight them to obtain an initial baseline association value; Identify the medical business scenario category that generated the data table, and match the corresponding adaptive time decay coefficient according to the scenario category, wherein the decay coefficient of the first scenario is greater than the decay coefficient of the second scenario. The adaptive time decay coefficient and the time difference of arrival are substituted into the natural exponential decay formula to generate a dynamic penalty term; Multiply the initial baseline association value by the dynamic penalty term, and output the decayed value as the edge weight of the business association graph.

5. A medical business data lake warehouse construction system based on a distributed architecture as described in claim 1, characterized in that, In the clustering module, splitting the associated clusters into synchronous micro-batches and generating synchronization parameters including synchronization order, concurrency, and batch data volume specifically includes the following steps: Based on the strong dependency directed edges between nodes in the business association graph, the data tables within the association cluster are topologically sorted to determine the synchronization order; Using the inverse of the resource conflict degree in the resource conflict graph as the constraint knapsack capacity, the sorted data table sequence is dynamically programmed and divided into multiple synchronous micro-batches. Based on the graph connectivity density of the nodes within a single micro-batch, the batch data volume is allocated in a positive direction and the concurrency is limited in a negative direction to generate synchronization parameters.

6. A medical business data lake warehouse construction system based on a distributed architecture as described in claim 5, characterized in that, The dynamic programming segmentation of the sorted data table sequence into multiple synchronous micro-batches specifically includes the following steps: Initialize the dynamic programming state matrix to record the target batch partitioning cost of the data table sequence at different cut points; For any candidate interval in the data table sequence, extract the peak edge weights of all data tables in the interval in the resource conflict graph, and determine the upper limit of the interval capacity of the candidate interval by the reciprocal of the peak edge weights; Calculate the cumulative data volume within the candidate interval. If the cumulative data volume exceeds the upper limit of the interval capacity, mark the candidate interval as unreachable in the state matrix. Based on the state transition equation, and under the premise of avoiding the unreachable state, the set of target cutting points that makes the number of global batch divisions meet the preset cost condition is solved, and synchronous micro-batches are output according to the set of target cutting points.

7. A medical business data lake warehouse construction system based on a distributed architecture as described in claim 1, characterized in that, In the synchronization module, the formation of the source endpoint, the synchronization task progress, and the database entry progress specifically include the following steps: Parse the redo log sequence number of the source database and record it as the source location; Extract the buffer queue consumption offset between the read thread and the write thread inside the data synchronization component and record it as the synchronization task progress; Obtain the global transaction timestamp of the distributed data lake warehouse cluster, extract the high watermark of committed transactions, and construct a status tracking link to form the data storage progress.

8. A medical business data lake warehouse construction system based on a distributed architecture according to claim 1, characterized in that, In the determination module, determining the completeness of the associated cluster specifically includes the following steps: Calculate the relative displacement difference between the source endpoint, the synchronization task progress, and the data drop progress, and map it to the data physical readiness rate; Extract the proportion of data in the micro-batch that is not matched due to missing foreign key dependencies, and map it to the business logic isolation rate; The completeness of the associated cluster is calculated by dynamically weighting and fusing the physical readiness rate and the reciprocal of the logical isolation rate.

9. A medical business data lake warehouse construction system based on a distributed architecture according to claim 1, characterized in that, In the optimization module, adjusting the synchronization parameters and performing local clustering of the associated clusters specifically includes the following steps: Calculate the joint gradient between the retry ratio of the distributed data lake warehouse transactions and the backlog, and dynamically increase or decrease the concurrency and batch data volume of subsequent micro-batches through a feedback controller; When a deviation that reaches the third preset threshold is identified, the data table that caused the deviation is locked, and a local subgraph within the associated cluster is extracted with the data table as the center. The overall topology of the non-subgraph portion within the original associated cluster remains unchanged. Only the local subgraph is re-performed with constraint clustering to generate updated synchronization parameters, which are then returned to the synchronization module.