A configuration-based medical health data collection pipeline method, system, and storage medium

Abstract

The application discloses a kind of based on configuration medical health data acquisition pipeline method, system and storage medium.The method comprises the following steps: receiving and parsing acquisition configuration, the acquisition configuration at least declares source database, target database, query statement and the corresponding relationship of source field and target field;Acquisition configuration is combined with this time running input parameter, generates immutable running snapshot, as the only basis of single execution;According to whether each configuration item in acquisition configuration exists, automatically assemble the step chain of fixed sequence, the step chain includes target cleaning, source query and target writing step, and optional step assembled according to configuration existence;Pipeline executor maintains unified execution context, executes each step according to step chain order, and writes data in source database into target database after conversion.The present application drives acquisition with a versionable declarative configuration, considers running snapshot audit, semi-structured data integrated processing, quality inspection line level isolation shunting and cross-database complement, reduces the repeated development amount of new data set, and is suitable for medical health data convergence scene.

Description

Technical Field

[0001] This invention relates to the field of medical information technology, and in particular to a configuration-based medical and health data acquisition pipeline method, system, and storage medium. Background Technology

[0002] In the construction of regional health platforms, hospital interconnection and smart health datasets, medical institutions need to aggregate various types of business data, such as outpatient and emergency medical records, medical record front pages, and laboratory reports, from source databases of different vendors and models, such as Oracle, DB2, and SQL Server, into a target database that conforms to standards and specifications.

[0003] In existing medical and health data collection scenarios, the source data generally has the following characteristics: (1) strong heterogeneity, the same business field may be distributed in multiple databases and tables, and cross-database association is required to complete it; (2) a lot of semi-structured data, such as large fields of electronic medical record XML / CLOB, JSON array of test details, and duplicate nodes of test report XML; (3) rapid standard evolution, with frequent addition of data items in national standards and industry datasets; (4) high quality requirements, local codes need to be mapped to standard codes, failure records need to be traceable and cannot block the writing of the entire batch of compliant data; (5) high requirements for operation and maintenance, it needs to support timed collection, supplementary collection, external system triggering, and tasks cannot be repeatedly executed or occupied for a long time under multi-instance deployment.

[0004] In existing technologies, data acquisition is typically achieved using ETL tools, stored procedures, data buses, or workflow engines. However, for each new type of business dataset, it is often necessary to develop separate extraction, transformation, and loading programs or rewrite stored procedures, resulting in a large amount of repetitive development. For semi-structured data such as electronic medical record XML and test result JSON arrays, scripts or sub-jobs need to be integrated in addition to ETL tools to complete the "expand first, then extract" process. The interconnection of multiple tools can lead to the risk of broken links and mismatched fields. When data quality verification fails, existing solutions cannot record the reason for the failure without blocking the writing of the entire batch of compliant data. In addition, capabilities such as cross-database data replenishment, runtime auditing, and multi-instance task scheduling also need to be customized separately.

[0005] Therefore, there is a need for a medical and health data acquisition technology solution that can be driven by a versionable declarative configuration, automatically generate a fixed-sequence collection step chain, and take into account runtime snapshot auditing, semi-structured integrated processing, quality inspection row-level isolation and diversion, and cross-database data supplementation. Summary of the Invention

[0006] To address the problems in existing technologies, such as the large amount of repetitive development required for new datasets, the need for multiple tools to be connected for semi-structured data, the easy blockage of writing compliant batches of data due to quality inspection failures, and the insufficient cross-database data replenishment and operational auditing capabilities, this invention provides a configuration-based medical and health data acquisition pipeline method, system, and storage medium.

[0007] In a first aspect, the present invention provides a configuration-based medical and health data acquisition pipeline method, comprising: a configuration receiving and parsing step: receiving acquisition configuration and parsing the acquisition configuration, wherein the acquisition configuration at least declares a source database, a target database, a query statement, and the correspondence between source fields and target fields; a snapshot solidification step: merging the parsed acquisition configuration with the input parameters of the current run to generate an immutable running snapshot, and using the immutable running snapshot as the sole basis for a single execution; a step chain assembly step: automatically assembling a step chain in a fixed order according to the existence of each configuration item in the acquisition configuration, wherein the step chain includes a target cleanup step, a source query step, and a target write step, and also includes optional steps assembled according to the existence of configuration items, wherein the optional steps include one or more of the following: a cross-source complement step, a row expansion step, a field extraction step, a quality verification step, and an isolation and diversion step; and a pipeline execution step: a pipeline executor maintains a unified execution context, executes each step in the order of the step chain, and acquires and transforms data from the source database based on the immutable running snapshot before writing it into the target database.

[0008] In one optional implementation, the data collection configuration further includes one or more of the following configuration items: data deletion strategy; cross-source data complementation configuration for supplementing columns from other data sources to the main query results; complex field extraction rules for extracting data from XML or JSON structured data to target fields; target table and primary key generation strategy; and standard code mapping rules for mapping local codes to standard codes.

[0009] In one optional implementation, the primary key generation strategy includes: when the collection configuration declares a set of primary key columns, and the value of a certain primary key column in a row is empty, the primary key is automatically filled with a sign according to the strategy; if a non-empty primary key already exists, the original value is used; the types of the strategy include: a host serial number strategy, which calls the unified serial number service of the deployment environment to generate the business primary key when the runtime environment provides serial number capability; a UUID strategy, which generates a UUID string when the runtime environment does not have a serial number service or the serial number service call fails; and a database auto-increment or sequence strategy, which is generated by the default value of the target database or generated by retrieving it after insertion.

[0010] In one optional implementation, the standard code mapping rules include: retrieving standard codes or standard names from the standard code mapping table according to dictionary type, local code, and optional institutional code; prioritizing institutional-level mapping during parsing, and falling back to global mapping if no match is found; if no match is found, marking the failure according to the rules, or using the default code allowed by the rules and recording the reason for the failure.

[0011] In one optional implementation, the immutable runtime snapshot includes a batch identifier, a service or dataset identifier, a configuration text or version reference, a parameter dictionary, and an optional rule payload; the rule payload includes a cleaning or verification rule text, a rule set identifier and version, and optional embedded standard code mapping auxiliary data; after the immutable runtime snapshot is fixed, retries, supplementary data collection, and queue consumption are all performed based on the same immutable runtime snapshot.

[0012] In one optional implementation, the fixed order of the step chain is as follows: target cleanup step, source query step, cross-source complementation step, row expansion step, field extraction step, quality verification step, isolation and diversion step, and target write step; wherein, the target cleanup step deletes old data in the target database according to the configuration; the source query step executes the query statement in the source database and loads the query results into a row set; the cross-source complementation step queries other data sources for each row in the row set according to the association key and merges the returned columns; the field extraction step parses and extracts JSON or XML data of the specified column and writes the extraction results into the target field slot; the quality verification step verifies row by row according to the rule set and performs standard code mapping, marks failed rows and records the reasons; the isolation and diversion step writes failed rows to the isolation area for persistent storage and removes them from the row set, only allowing rows that pass the verification to enter the target write step; the target write step writes to the target table in batches according to the field mapping.

[0013] In one optional implementation, the row expansion step includes: expanding the JSON or XML array in the parent row record into multiple sub-rows, each sub-row inheriting the key fields of the parent row to form a sub-row set; the field extraction step includes: performing path operations on specified columns in the sub-row set according to XPath or JSON Path, and writing the extraction results into target field slots, wherein the extraction results are decoupled from the original large field; the target writing step includes: reading values ​​from the target field slots and writing them into target table columns according to field mapping.

[0014] In an optional implementation, the method further includes a task scheduling and queue execution step: persisting the collection request as a queue task, the queue task including an idempotent key, priority, planned execution time, lease lock, and fixed parameter snapshot; the consumer preempts the task through condition updates and sets the lease expiration time, and expired unfinished tasks are reclaimed; after successful preemption, the immutable running snapshot is constructed and the pipeline execution step is invoked.

[0015] Secondly, the present invention provides a configuration-based medical and health data acquisition pipeline system, comprising: a configuration parsing module, used to receive and parse the acquisition configuration, wherein the acquisition configuration at least declares a source database, a target database, a query statement, and the correspondence between source fields and target fields; a snapshot persistence module, used to merge the parsed acquisition configuration with the input parameters of the current run to generate an immutable running snapshot, and use the immutable running snapshot as the sole basis for a single execution; a step chain assembly module, used to automatically assemble a step chain in a fixed order according to the existence of each configuration item in the acquisition configuration, wherein the step chain includes a source query step and a target write step, and also includes optional steps assembled according to the existence of configuration items; and a pipeline execution module, used to maintain a unified execution context, execute each step in the order of the step chain, and collect and transform data from the source database and write it to the target database based on the immutable running snapshot.

[0016] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the configuration-based medical and health data acquisition pipeline method described in the first aspect above.

[0017] Compared with the prior art, the beneficial effects of the present invention include: (1) Low implementation cost, new datasets are mainly configured, reducing redundant development. Existing solutions often require separate development of extraction, transformation, loading programs or rewriting of stored procedures for each new type of business dataset. This invention converges "what to look for, how to map, whether to expand, whether to extract, and whether to perform quality inspection" into a persistent configuration. The step chain is automatically assembled according to the existence of configuration items. Business changes are mainly modified by changing the configuration, avoiding the need to repeatedly write pipeline skeleton code and multiple sets of scheduling glue logic.

[0018] (2) Adapting to semi-structured data, row expansion and path extraction are completed within the same pipeline. Medical source data often takes the form of "a row containing large fields such as JSON of test details or XML of medical records". Existing solutions require scripts or sub-jobs outside of ETL tools to complete "expansion first, then extraction". This invention arranges row expansion and field extraction into adjacent steps in a fixed order. The field extraction step writes the extraction results into the target field slot, and the target writing step writes the results into the database from the slot. This allows semi-structured processing and structured mapping to share the same row set and the same configuration document, eliminating the need for multiple tools to be connected in series and reducing the risk of link breakage and field mismatch.

[0019] (3) Cross-database capability is built-in, supporting heterogeneous source assembly. In the collection configuration, the source database and the target database can be identified as different databases; the cross-source complement step merges columns from other data sources to the row set according to the association key, and multiple rows can be configured; the pipeline execution step registers multiple data source connections according to the configuration, and routes them uniformly in the execution context.

[0020] (4) The operation is reproducible and auditable, and the quality inspection does not block the entry of compliant data into the database. The snapshot solidification step generates an immutable running snapshot. Retry, supplementary collection, and queue consumption are all based on the same snapshot, which makes it reproducible and auditable. The rows that fail the quality verification are marked and written to the isolation area for persistent storage. Only the rows that pass the verification enter the target writing step. The failure record is traceable and does not block the writing of the entire batch of compliant data. Attached Figure Description

[0021] Figure 1 A flowchart of pipeline steps provided for embodiments of the present invention; Figure 2 This is a data flow diagram for extracting complex fields provided in an embodiment of the present invention.

[0022] Figure labeling: 11—Configuration parsing; 12—Snapshot persistence; 13—Step chain assembly; 31—Target cleanup; 32—Source query; 33—Cross-source data complement; 34—Row expansion; 35—Field extraction; 36—Quality verification; 37—Isolation and diversion; 38—Target writing; 41—Pipeline execution; 42—Execution result audit log; 51—Queue scheduling; 101—Parent row record; 102—Row expansion module; 103—Child row set; 104—Field extraction module; 105—Target field slot; 106—Database mapping module; 107—Target table column. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] Example 1: A Configuration-Based Healthcare Data Acquisition Pipeline Method like Figure 1 As shown, this embodiment provides a configuration-based medical and health data acquisition pipeline method, including configuration parsing S1 (reference numeral 11), snapshot solidification S2 (reference numeral 12), step chain assembly S3 (reference numeral 13), pipeline execution S4 (reference numeral 41), and optional queue scheduling S5 (reference numeral 51). Pipeline execution S4 outputs an execution result audit log 42. An external system triggers a data acquisition request to be enqueued, which then proceeds to snapshot solidification S2 after queue scheduling S5.

[0025] S1. Configure receiving and parsing steps. Receive and parse the acquisition configuration. The acquisition configuration should include at least the following: 1.1 Configure the source database and target database. The source database and target database can be identified as different databases; 1.2 Configure the query statement and the binding relationship of runtime parameters; 1.3 Configure the correspondence between source fields and target fields. The correspondence may include constants, multi-source selection, and simple type conversion. 1.4 Configure data deletion policy; 1.5 Configure cross-source complement: Columns can be supplemented from other data sources in the main query results; 1.6 Configure complex field extraction rules: Extract data from structured data types such as XML or JSON to the target field; 1.7 Configure Target Table and Primary Key Generation Strategy: When a primary key column set is declared in the configuration, and the value of a primary key column in a row is empty, a filler sign is automatically added according to the strategy; if a non-empty primary key already exists, it is used. Supported strategy types include: (a) Host Serial Number: When a primary key column is declared in the configuration and the runtime environment provides serial number capability, the unified serial number or sequence service of the deployment environment is called to generate the business primary key; (b) UUID: When the host does not have a serial number service or the call fails, a connectionless character or standard UUID string is generated; (c) Database Auto-increment or Sequence: When the target table is designed to be auto-increment or sequence and the configuration does not force the filler sign on the reference side, it is generated from the default value of the target database or retrieved after insertion; 1.8 Configure Standard Code Mapping: The mapping source is the standard code mapping table maintained by the metadata library or the target side. Standard codes or standard names are retrieved by dictionary type, local code, or optional organization code. Mapping actions are declared in the cleaning rules, such as specifying the fields to be mapped and the dictionary type. Rules can be versioned and reused for multiple collection items. The parsing priority is organization-level mapping first; if no match is found, the global mapping is rolled back; if still no match is found, the rule marks it as a failure, or the default code allowed by the rule is used and the reason for failure is recorded. Implementers add, delete, and modify the mapping relationship between local codes and standard codes in the mapping table. The collection configuration only references dictionary types and fields, and does not write the full mapping table into the business configuration. When strong auditing is required, a subset of the mappings used in this instance can be included in the rule payload.

[0026] S2, Snapshot Fixing Step. The data collection configuration from step S1 is merged with the input parameters for this run (institution, visitor identifier, service order number, etc.) to generate an immutable runtime snapshot, serving as the sole basis for each execution. The snapshot includes: batch identifier, service or dataset identifier, configuration text or version reference, parameter dictionary, and optional rule payload. The rule payload is a combined payload of the validation rule set and transformation-type cleaning actions, written together during snapshot fixing for quality governance purposes, ensuring that quality inspection does not rely on "reading the latest rules during execution." The rule payload may include at least the following: (a) Cleaning or validation rule text: an ordered list of actions, such as required fields, regular expressions, value ranges, whitespace removal, default values, and standard code mapping actions. Standard code mapping actions are used to write local codes back to standard codes, and a flag is added if the action fails; (b) Rule set identifier and version: rule set ID, version number, or content hash for auditing purposes; (c) Standard code mapping auxiliary data (optionally embedded): a subset of "dictionary type × local code → standard code" required for this batch run, or a read-only snapshot pointing to the mapping table at the time of solidification. If not embedded, the quality inspection steps will query the mapping table during execution according to the rule actions in the snapshot, but the rule actions themselves are still based on the text in the snapshot. The rule payload does not include source SQL, field mapping, row expansion, path extraction, and other collection structure configurations, which are already in the configuration text. The only consumption position of the rule payload in the step chain is the failure tagging result on which the quality verification step and its associated isolation and diversion depend. After the snapshot is solidified, retries, supplementary collection, and queue consumption are all based on the same snapshot, achieving reproducibility and auditability. When multiple data collection items are executed together, an independent intermediate state namespace is allocated to each subtask to avoid data overwriting between steps.

[0027] S3, Step Chain Assembly Steps. Whether each sub-step is assembled depends on the existence of the configuration item. The order of the step chain is fixed and cannot be arbitrarily changed. For example... Figure 1 The detailed step chain is shown in the middle, and the step chain is as follows: 3.1 Target Cleanup (Optional): Delete old data in the target database according to the configuration. The parameters are derived from the runtime snapshot. 3.2 Source Query: Execute the query in the source database, and load the query results as a row collection; large objects can be converted into parsed documents; 3.3 Cross-source complement (optional, multiple rows allowed): Query each row of the row set using the join key in other data sources and merge the returned columns; 3.4 Line expansion (optional): e.g. Figure 2 As shown, the parent row record 101 (1 row) is expanded into multiple rows by the row expansion module 102, forming a child row collection 103 (N rows), and each child row inherits the key fields of the parent row; 3.5 Field Extraction (Optional): such as Figure 2 As shown, the field extraction module 104 performs path calculations on complex data such as JSON or XML in the specified column using XPath or JSON Path, and writes the extraction results into the target field slot 105, decoupling the extraction results from the original large field. 3.6 Quality verification (optional): Verify the mapping between each line of the rule set and the standard code, mark the failed lines and record the reasons; 3.7 Isolation and Diversion (Optional): Failed rows are written to the metadata isolation area for persistent storage and removed from the row set. Only rows that fail to pass through the write step are allowed to proceed. 3.8 Target Write: Batch write to the target table by field mapping; null primary key values ​​can be automatically assigned serial numbers according to a strategy; for example... Figure 2 As shown, the database mapping module 106 reads the value from the target field slot 105 and writes it to the target table column 107.

[0028] S4. Pipeline Execution Steps. The pipeline executor executes the steps sequentially according to the chain of steps assembled in step S3: maintaining a unified execution context, which includes runtime snapshots, multiple data source connections, and row sets; recording the status and time consumption of each step; outputting the execution results; multiple collection items can be executed serially in the same transaction for multiple sub-pipelines.

[0029] S5. Task Scheduling and Queue Execution Steps (Optional). Collection requests are persisted as queued tasks, each containing an idempotent key, priority, planned execution time, lease lock, and a snapshot of persistent parameters. Consumers preempt tasks based on updated conditions and set lease expiration times; expired unfinished tasks can be reclaimed to prevent multiple instances from becoming stuck. After successful preemption, a snapshot from step S2 is constructed, and step S4 is called. Entry points such as timed events and interface enqueues all follow the principle of "queuing on the control plane and unified pipeline execution on the execution plane."

[0030] Example 2: Application Case of Inspection Report Data Collection Taking examination report data collection as an example: Configure the source database as the hospital's business Oracle database (including inpatient, examination, and patient master indexes, etc.), and the target database as the regional health platform standard database; configure the main query statement to query examination reports and related patient information by the visit serial number; configure complex field extraction rules to extract examination findings, image links, and report links from the large XML field of the examination report body in the report row using multiple XPaths; configure field mapping and time normalization rules to write examination time, report time, etc., into standard table fields; and configure a quality verification rule set to perform standard code mapping verification for document type, examination item code, etc.

[0031] During execution, the system permanently stores an immutable snapshot containing parameters such as institution and visitor identifier (e.g., inpatient registration number). Based on configuration availability, it automatically assembles a chain of steps: "Target Cleanup → Source Query → Field Extraction → [Quality Verification → Isolation and Diversion] → Target Writing." The target cleanup step deletes old examination report data on the target side based on the visitor registration number. The source query step executes a main query to obtain a set of report rows and converts large XML fields into parsable text. The field extraction step performs path operations such as / examination / findings / text(), / examination / image_url / text(), and / examination / examination_url / text() on the same XML text, writing the results to target field slots such as Examination Findings, Image Links, and Report Links. Other structured columns (examination conclusion, examination site, reporting department, reporter, reviewer, hospital project code name, etc.) are directly filled into the corresponding slots according to the mapping. If quality verification is performed, failed rows are written to the metadata isolation area with the reason recorded, and only batch-written to the target table HT_EXAM_REPORT_MAIN. When retrying or supplementing the snapshot, the queue consumers re-execute based on the same snapshot to ensure consistent results.

[0032] Example 3: Configuration-Based Healthcare Data Acquisition Pipeline System This embodiment provides a configuration-based medical and health data acquisition pipeline system, including a configuration parsing module, a snapshot persistence module, a step chain assembly module, and a pipeline execution module. The configuration parsing module is used to execute step S1; the snapshot persistence module is used to execute step S2; the step chain assembly module is used to execute step S3; and the pipeline execution module is used to execute step S4. Optionally, the system further includes a queue scheduling module for executing step S5. The specific implementation of each module is the same as the corresponding steps in Embodiment 1, and will not be repeated here.

[0033] Example 4: Computer-readable storage medium This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the configuration-based medical and health data acquisition pipeline method described in Embodiment 1 above.

[0034] Those skilled in the art can implement some functions of data acquisition through one or more alternative methods based on existing ETL, stored procedures, buses, workflows, and other known means. However, if the goal of this invention is to simultaneously achieve configurability, auditability, semi-structured integrated processing, non-blocking of compliant data entry during quality inspection, and scalable scheduling, the overall solution disclosed in this application, which features configuration-driven fixed-step pipelines, runtime snapshots, and isolated and diverted quality inspection, is adopted. This solution has less customized development, a more complete audit loop, and is more suitable for the operation of medical batch data acquisition.

[0035] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A configuration-based medical and health data acquisition pipeline method, characterized in that, include: Configuration receiving and parsing steps: Receive the collection configuration and parse the collection configuration. The collection configuration at least declares the source database, target database, query statement, and the correspondence between source fields and target fields. Snapshot solidification step: The parsed acquisition configuration is merged with the input parameters of this run to generate an immutable running snapshot, and the immutable running snapshot is used as the sole basis for a single execution; Step chain assembly steps: Based on the existence of each configuration item in the collection configuration, a fixed sequence of steps is automatically assembled. The step chain includes a source query step and a target writing step, as well as optional steps assembled based on the existence of configuration items. The optional steps include one or more of the following: target cleanup step, cross-source data completion step, row expansion step, field extraction step, quality verification step, and isolation and diversion step. Pipeline execution steps: The pipeline executor maintains a unified execution context, executes each step in the order of the step chain, and collects and transforms data from the source database based on the immutable runtime snapshot before writing it into the target database.

2. The configuration-based medical and health data acquisition pipeline method according to claim 1, characterized in that, The data collection configuration also includes one or more of the following configuration items: data deletion strategy; cross-source data completion configuration, used to supplement columns from other data sources to the main query results; complex field extraction rules, used to extract data from XML or JSON structured data to the target field; target table and primary key generation strategy; Standard code mapping rules are used to map local codes to standard codes.

3. The configuration-based medical and health data acquisition pipeline method according to claim 2, characterized in that, The primary key generation strategy includes: when the collection configuration declares a set of primary key columns, and the value of a primary key column in a row is empty, the column is automatically filled with a sign according to the strategy; if a non-empty primary key already exists, the original value is used. The types of the strategy include: host serial number strategy, which calls the unified serial number service of the deployment environment to generate the business primary key when the runtime environment provides serial number capability; UUID strategy, which generates a UUID string when the runtime environment does not have a serial number service or the serial number service call fails; and database auto-increment or sequence strategy, which is generated by the default value of the target database or retrieved after insertion.

4. The configuration-based medical and health data acquisition pipeline method according to claim 2, characterized in that, The standard code mapping rules include: retrieving standard codes or standard names from the standard code mapping table according to dictionary type, local code, and optional institutional code; prioritizing institutional-level mapping during parsing, and falling back to global mapping if no match is found; if still no match is found, marking it as a failure according to the rules, or using the default code allowed by the rules and recording the reason for the failure.

5. The configuration-based medical and health data acquisition pipeline method according to claim 1, characterized in that, The immutable runtime snapshot includes a batch identifier, a service or dataset identifier, a configuration text or version reference, a parameter dictionary, and optional rule payloads; the rule payloads include a cleaning or verification rule text, a rule set identifier and version, and optional embedded standard code mapping auxiliary data; after the immutable runtime snapshot is solidified, retries, supplementary data collection, and queue consumption are all performed based on the same immutable runtime snapshot.

6. The configuration-based medical and health data acquisition pipeline method according to claim 1, characterized in that, The fixed order of the steps is as follows: target cleanup step, source query step, cross-source complement step, row expansion step, field extraction step, quality verification step, isolation and diversion step, and target write step. Specifically, the target cleanup step deletes old data in the target database according to the configuration; the source query step executes the query statement in the source database and loads the query results into a row set; the cross-source complement step queries other data sources for each row in the row set according to the association key and merges the returned columns; the field extraction step parses and extracts JSON or XML data for specified columns and writes the extraction results into the target field slots; the quality verification step verifies row by row according to the rule set and performs standard code mapping, marking failed rows and recording the reasons; the isolation and diversion step writes failed rows to an isolation area for persistent storage and removes them from the row set, only allowing rows that pass verification to enter the target write step; and the target write step writes data to the target table in batches according to the field mapping.

7. The configuration-based medical and health data acquisition pipeline method according to claim 6, characterized in that, The row expansion step includes: expanding the JSON or XML array in the parent row record into multiple sub-rows, with each sub-row inheriting the key fields of the parent row to form a sub-row set; the field extraction step includes: performing path operations on the specified columns in the sub-row set according to XPath or JSON Path, and writing the extraction results into the target field slot, wherein the extraction results are decoupled from the original large field; the target writing step includes: reading the value from the target field slot and writing it into the target table column according to the field mapping.

8. The configuration-based medical and health data acquisition pipeline method according to any one of claims 1 to 7, characterized in that, It also includes task scheduling and queue execution steps: persisting the collection request as a queue task, the queue task includes an idempotent key, priority, planned execution time, lease lock and fixed parameter snapshot; the consumer preempts the task through condition update and sets the lease expiration time, and expired unfinished tasks are reclaimed; after successful preemption, the immutable running snapshot is constructed and the pipeline execution steps are called.

9. A configuration-based medical and health data acquisition pipeline system, characterized in that, include: The configuration parsing module is used to receive the collection configuration and parse the collection configuration. The collection configuration at least declares the source database, the target database, the query statement, and the correspondence between the source fields and the target fields. The snapshot persistence module is used to merge the parsed acquisition configuration with the current run input parameters to generate an immutable running snapshot, and use the immutable running snapshot as the sole basis for a single execution. The step chain assembly module is used to automatically assemble a step chain in a fixed order based on the existence of each configuration item in the acquisition configuration. The step chain includes a target cleanup step, a source query step, and a target write step, and also includes optional steps assembled based on the existence of configuration items. The pipeline execution module is used to maintain a unified execution context, execute each step in the order of the step chain, and collect and transform data from the source database based on the immutable runtime snapshot before writing it into the target database.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the configuration-based medical and health data acquisition pipeline method according to any one of claims 1 to 8.

Images