Medical data reconstruction method, system and equipment based on index alignment and medium
By using an index-aligned approach, medical data is reconstructed into time and event data tables, and explicit indexes are built and linked. This solves the problems of slow data processing speed and memory overflow in traditional methods, and achieves efficient and scalable medical data analysis.
Patent Information
- Application Number
- CN202610120059.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-28
- Publication Date
- 2026-04-28
AI Technical Summary
When processing longitudinal medical data from large biobanks, existing technologies cannot efficiently establish logical relationships between time and events, leading to performance bottlenecks and memory overflow risks, making it difficult to meet the needs of efficient and accurate medical data analysis.
Using an index-aligned approach, medical data is reconstructed into time-based and event-based data tables. Explicit indexes are generated and linked, including the construction of a sequence index for the time-based data table and the generation of a continuous sequence index for the event-based data table. Hash functions are then used for fast matching and linking.
It significantly improves processing speed, eliminates the problem of temporary table explosion, improves memory efficiency, has high scalability, can efficiently process tens of millions to hundreds of millions of records, and is suitable for various vertical medical data reconstruction scenarios.
Smart Images
Figure CN121935252A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data processing technology, specifically relating to a method, system, device, and medium for medical data reconstruction based on index alignment. Background Technology
[0002] Currently, large biobanks (such as UK Biobank) often employ a non-relational strategy to improve storage efficiency, physically separating and storing longitudinal medical data (such as multiple diagnosis times and corresponding disease codes) for the same entity. Time data is recorded in multi-column wide tables, while event codes are often encoded in a single field containing a delimiter. This storage model leads to a break in the logical relationship between time and events. Traditional data processing methods rely on row-by-row parsing and multi-level nested loops, which not only fail to efficiently establish explicit relationships but also generate serious performance bottlenecks and memory overflow risks when dealing with massive amounts of data, making it difficult to meet the needs of efficient and accurate medical data analysis. Summary of the Invention
[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides a medical data reconstruction method, system, device and medium based on index alignment to solve the above-mentioned technical problems.
[0004] In a first aspect, the present invention provides a medical data reconstruction method based on index alignment, comprising: For the source data table, multiple time field values corresponding to the same entity identifier are converted into a time data table containing the entity identifier, time value, and time sequence identifier; at the same time, field values containing multiple event codes corresponding to the same entity identifier are split into event data tables containing the entity identifier and a single event code. For the time data table, the time sequence identifier is directly used as the sequence index; for the event data table, a continuous sequence index is generated for each event record based on the original order of the event codes under each entity identifier. By associating records with the same entity identifier and the same sequence index value in the time data table and the event data table, the reconstructed medical data is obtained.
[0005] In an optional implementation, multiple time field values corresponding to the same entity identifier are converted into a time data table containing the entity identifier, time value, and time sequence identifier, including: Perform a reverse pivot operation on multiple time fields representing the same time dimension in the source data table, convert the field name of each time field into the time sequence identifier, and use the corresponding field value as the time value to generate the time data table.
[0006] In an optional implementation, converting the field name of each time field into the time sequence identifier includes: Extract the numeric suffix from the field name and use the numeric suffix as the time sequence identifier for the corresponding record.
[0007] In an optional implementation, the field value containing multiple event codes corresponding to the same entity identifier is split into an event data table containing the entity identifier and a single event code, including: The field value is split into multiple independent event codes based on a preset delimiter; Each extracted event code is associated with its corresponding entity identifier to form an event record in the event data table.
[0008] In an optional implementation, a sequential index is generated for each event record based on the original order of event codes under each entity identifier, including: In the event data table, for each entity identifier group, a window function is applied to generate consecutive serial numbers, and these serial numbers are used as sequence indexes; The window function is the ROW_NUMBER() function, which groups entities by entity identifier using the PARTITIONBY clause and performs stable sorting based on the order of event codes in the original field values using the ORDERBY clause.
[0009] In an optional implementation, for the event data table, based on the original order of event codes under each entity identifier, a continuous sequence index is generated for each event record, including: The event records under each entity identifier group are distributed to multiple parallel processing units; wherein, each event record in the event data table corresponds to one entity identifier and one event code obtained after splitting. Within each of the parallel processing units, a local sequential number starting from zero is generated for the event records assigned to it; The global starting offset of the record processed by each of the parallel processing units is determined through a parallel, hierarchical aggregation computation. The local consecutive number of each event record is added to the global starting offset of its corresponding parallel processing unit to obtain the final consecutive sequence index of the event record.
[0010] In an optional implementation, the time data table and the event data table are associated with records having the same entity identifier and the same sequence index value to obtain reconstructed medical data, including: Using the time data table as the construction table, based on the composite primary key composed of the entity identifier and the sequence index, the hash value is calculated by a hash function, and the corresponding record is stored in the corresponding position of the memory hash table; Using the event data table as a probe table, traverse each record, calculate the hash value based on the same composite primary key, and search for the time record with the same composite primary key value in the memory hash table. Then, associate the found time record with the event record. Merge each successfully associated time record and event record, and output a reconstructed record containing entity identifier, time value, event code and sequence index.
[0011] Secondly, the present invention provides a medical data reconstruction system based on index alignment, comprising: The field splitting module is used to convert multiple time field values corresponding to the same entity identifier into a time data table containing the entity identifier, time value, and time sequence identifier for the source data table; at the same time, it splits the field values containing multiple event codes corresponding to the same entity identifier into an event data table containing the entity identifier and a single event code. The index building module is used to directly use the time sequence identifiers in the time data table as sequence indexes; and to generate a continuous sequence index for each event record based on the original order of the event codes under each entity identifier in the event data table. The data reconstruction module is used to associate records with the same entity identifier and the same sequence index value in the time data table and the event data table to obtain the reconstructed medical data.
[0012] Thirdly, a device is provided, comprising: Memory for storing index-aligned medical data reconstruction procedures; A processor, used to implement the steps of the index-aligned medical data reconstruction method as provided in the first aspect when executing the index-aligned medical data reconstruction procedure.
[0013] Fourthly, a computer-readable medium is provided, on which an index-aligned medical data reconstruction program is stored, wherein when executed by a processor, the index-aligned medical data reconstruction program implements the steps of the index-aligned medical data reconstruction method provided in the first aspect.
[0014] The beneficial effects of this invention are that the index-aligned medical data reconstruction method, system, device, and medium provided by this invention, through vectorized processing and explicit index construction, completely avoid traditional row-by-row operations and nested loops, increasing processing speed by tens of times. Simultaneously, the index alignment mechanism eliminates the temporary table explosion problem, significantly improving memory efficiency. The solution is highly scalable, capable of efficiently processing tens of millions to hundreds of millions of records, and is applicable to various vertical medical data reconstruction scenarios such as drug prescriptions and surgical records, providing general and efficient basic technical support for large-scale medical data analysis. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.
[0017] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.
[0018] Figure 3 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation
[0019] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
[0021] The index-aligned medical data reconstruction method provided in this embodiment of the invention is executed by a computer device, and correspondingly, the index-aligned medical data reconstruction system runs on the computer device.
[0022] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be an index-aligned medical data reconstruction system. Depending on different needs, the order of the steps in this flowchart can be changed, and some can be omitted.
[0023] like Figure 1 As shown, the method includes: S1. For the source data table, convert multiple time field values corresponding to the same entity identifier into a time data table containing the entity identifier, time value, and time sequence identifier; at the same time, split the field values containing multiple event codes corresponding to the same entity identifier into an event data table containing the entity identifier and a single event code; S2. For the time data table, the time sequence identifier is directly used as the sequence index; for the event data table, a continuous sequence index is generated for each event record based on the original order of the event codes under each entity identifier. S3. Associate the records with the same entity identifier and the same sequence index value in the time data table and the event data table to obtain the reconstructed medical data.
[0024] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.
[0025] S101. Convert multiple time field values corresponding to the same entity identifier into a time data table containing the entity identifier, time value, and time sequence identifier, including: First, the system reads the source data table stored in the database or data file. This source data table contains an "Entity Identifier (EID)" field to uniquely identify a sample or individual, and a series of time fields to store multiple records at the same time dimension. For example, columns with field names following the pattern "p41280_a0", "p41280_a1", ..., "p41280_aN", where the numeric suffixes (0,1,...,N) implicitly represent the time sequence or instance index.
[0026] The system automatically identifies these time field sets that share a common prefix and numeric suffix pattern. Then, it performs an unpivot operation on these fields. This operation converts these column names (field names) into row data. Specifically, for each row in the source table (corresponding to one EID), the unpivot operation generates a new record for each non-nullable time field value. In the newly generated records: The "Entity Identifier (EID)" remains unchanged and is derived from the original line.
[0027] The "time value" comes from the cell value of the original time field.
[0028] The "time sequence identifier" is obtained by processing the field name of the original time field. In practice, the system applies string processing functions to the field name, such as using the regular expression _a(\d+)$ to match and extract the numeric suffix after the underscore _a. The extracted integer value (such as "3" extracted from "p41280_a3") is then directly assigned as the "time sequence identifier" of the current record.
[0029] Through the above operations, the data originally stored in a wide table format, with one EID corresponding to multiple columns of time values, has been transformed into a long table structured time data table. This table contains at least three columns: EID, date (time value), and idx (time sequence identifier). The idx column is derived directly from the numeric suffix of the field name, providing an explicit, numerical sequence key for subsequent precise alignment.
[0030] Variant implementation: In some big data processing frameworks (such as Apache Spark), reverse pivoting can be implemented using the stack or melt function, which is logically equivalent to the SQL UNPIVOT function. Extracting the sequence identifier from field names can also be achieved using string splitting functions (such as splitting based on a specific delimiter "_a" and then extracting the second part), achieving the same technical effect as regular expression extraction.
[0031] S102. Split the field values containing multiple event codes corresponding to the same entity identifier into an event data table containing the entity identifier and a single event code, including: The system reads fields from the source data table that contain multiple event codes. A typical characteristic of such fields is that their values are long strings connected by a preset separator (such as a vertical bar "|"). For example, a field representing a disease diagnosis code might have a value in the form of "C34.1|I25.1|J18.9", indicating different disease codes recorded for the same individual in multiple medical treatments.
[0032] The splitting process first identifies and locates such fields. For each row in the table, the system retrieves the string value of that field based on its Entity Identifier (EID). Then, the splitting operation is performed: a string processing function (such as the .split('|') method in Python, the STRING_SPLIT function in SQL, or the split function in Spark) is called to split the long string into a list of event codes (such as ['C34.1', 'I25.1', 'J18.9']) using the delimiter "|" as the boundary.
[0033] Next, the system performs an Explode or Flatten operation, expanding each element in the list into a separate row. During this process, the system preserves the original entity identification relationships. Specifically, for each individual event code separated from the list, a new event record is generated. This record contains at least two key pieces of information: EID: The entity identifier of the record before it was split.
[0034] event_code: Taken from a single element in the event code list.
[0035] Finally, all the newly generated records are compiled into an event data table. The table is in long table format, with each row representing an independent event (event_code) experienced by an entity (EID), thus preparing the data for subsequent alignment with the time series.
[0036] Variant implementation methods and optimizations: Preservation of order: During the burst operation, the original position (such as an index starting from 0) of each event code in the list can be recorded as a temporary column for use in subsequent steps when the original input order needs to be strictly preserved.
[0037] Null value and exception handling: During implementation, the system can be configured to automatically filter out empty strings or invalid codes generated after splitting, ensuring the cleanliness of the event data table.
[0038] Performance optimization: For massive amounts of data, this splitting and bursting operation can be executed in parallel in a distributed computing framework (such as Apache Spark) and can be processed efficiently through the flatMap or explode function, avoiding single-point memory bottlenecks.
[0039] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.
[0040] S201. For the time data table, the time sequence identifier is directly used as the sequence index.
[0041] The input for this step is the time data table generated in step S101. This table contains at least three core fields: Entity Identifier (EID), Time Value (date), and Time Sequence Identifier (idx_time). The Time Sequence Identifier (idx_time) is a numeric suffix (e.g., 3) extracted directly from the time field name (e.g., p41280_a3) in the source table during the reverse pivoting phase. It is essentially an integer sequence representing the order of the original time column.
[0042] During processing, the system reads each row of records from the time data table. For each record, the system directly assigns the value of its "Time Sequence Identifier (idx_time)" field to a new field called "Sequence Index (index)" or uses it directly as a key for subsequent associations. This process is a simple field mapping or renaming operation, without involving any arithmetic operations, sorting, or renumbering.
[0043] For example, if an original record is (EID:123456, date:'2020-01-15', idx_time:3), after this step, the key used for association of this record will be (EID:123456, index:3). Here, the index value of 3 comes directly from the idx_time value of 3.
[0044] The output is a temporal data view that implicitly or explicitly includes a "sequence index". The technical advantage lies in explicitly materializing the sequence information implicit in the storage schema, expressed through field names, into a numeric key that can be directly used for equi-associations, laying a solid foundation for subsequent precise alignment with the event data table.
[0045] Variant implementation method: In actual programming, this step can be achieved through simple column selection or column renaming operations (e.g., in Pandas, df_time['index']=df_time['idx_time'], or in SQL, SELECT EID,date,idx_timeASindexFROMtime_table).
[0046] In high-performance scenarios that emphasize zero-copy, the reference to the time sequence identifier field can be used directly as the association key without creating a new data copy in memory.
[0047] The logic of this step differs fundamentally from that of the step "Generate a continuous index for the event data table" (S202): the former directly references existing explicit information, while the latter recalculates and generates a continuous sequence. This difference is one of the key design considerations for achieving efficient alignment.
[0048] S202. For the event data table, generate a continuous sequence index for each event record based on the original order of the event codes under each entity identifier.
[0049] First implementation method (based on declarative window functions): This method is suitable for execution in relational databases or big data engines that support SQL semantics (such as SparkSQL). The input is the event data table generated in step S102, which contains at least two columns: Entity Identifier (EID) and Event Code (event_code), and multiple records under the same EID have had their physical storage order in the original field values preserved through a splitting operation.
[0050] The system executes a window function query on this table. Specifically, it uses the ROW_NUMBER() function and specifies grouping by entity identifier using the PARTITIONBYEID clause. To preserve the original order of event codes, the ORDERBY clause can perform a stable sort based on an implicit original order identifier (such as the original position preserved during splitting). If this identifier does not exist, ORDERBY(SELECT NULL) maintains the existing physical order of records in the table (i.e., the default order after splitting, which is usually consistent with the original order). Finally, a consecutive row number starting from 1 is generated for each record within each EID group, and this row number is subtracted by 1 (or a sequence number starting from 0 is directly generated) to form a sequence index.
[0051] For example, executing the SQL statement: `SELECT EID, event_code, ROW_NUMBER() OVER(PARTITIONBYEIDORDERBYoriginal_position) - 1 AS index FROM event_table` will output an event data view containing a continuous sequence index that can be used for alignment.
[0052] Second implementation method (based on parallel computing framework): This approach is suitable for processing extremely large datasets in distributed or parallel computing environments (such as Apache Spark and Dask) to optimize performance. First, the event data table is hash-partitioned by entity identifier (EID), ensuring that all records with the same EID reside on the same compute node or within the same partition. Each partition can be further divided into multiple data blocks, processed by different parallel processing units (such as CPU cores or threads).
[0053] The process is divided into three stages: Local numbering stage: Each processing unit independently generates consecutive local numbers starting from 0 for the record blocks assigned to it that belong to the same EID, in the original order.
[0054] Global offset aggregation phase: A parallel prefix sum (ParallelPrefixSum) algorithm is used to efficiently calculate the starting offset of the record block processed by each processing unit in the global scope, using a tree-like or hierarchical communication method. For example, the number of records reported by each unit is used to calculate the total number of records processed by all previous units through prefix sum, which is the offset of that unit.
[0055] Global index synthesis phase: Each processing unit adds the local number of each record it generates to the global starting offset of its own unit to obtain the final, globally continuous sequence index of that record.
[0056] This approach avoids global sorting and data skew through parallel scanning and computation, achieving a linear time complexity of O(N). It also fully utilizes the SIMD instruction set of modern CPUs for vectorized accumulation, significantly improving the index generation efficiency for massive amounts of data.
[0057] Technical benefits: Regardless of the implementation method, each record in the event data table ultimately obtains a unique, continuous sequence index. This index is numerically isomorphic to the time sequence identifier in the time data table, thus creating conditions for subsequent precise association and alignment based on (EID, index). The choice of implementation method can be flexibly adapted according to the data scale, system environment, and performance requirements.
[0058] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.
[0059] S301. Construct a hash table: This step uses the time data table as the build table, aiming to create a fast-searchable data structure in memory. The system iterates through each row of the time data table, each record containing an entity identifier (EID), a time value (date), and a sequence index (index). The system uses a composite primary key consisting of (EID, index) as the key of the hash table.
[0060] The specific steps are as follows: Calculate the hash value: Call a predefined hash function, take the composite primary key as input, and output an integer hash value to determine the bucket position in the hash table. An efficient implementation example is: hash_key=((eid<<16)|index)%TABLE_SIZE, which combines the key and value through bitwise operations and modulo, aiming to reduce collisions and ensure even distribution.
[0061] Record storage: The complete original time record (or at least a pointer / reference containing the time value) is stored in the calculated hash table location. To handle hash collisions, linear probing (sequentially searching for the next empty slot) or chaining (maintaining a linked list in each bucket) can be used. The system typically monitors the hash table's load factor and dynamically resizes and rehashes all entries when it exceeds a threshold (e.g., 0.7) to maintain performance.
[0062] S302. Detection and Correlation: This step uses the event data table as a probe table, aiming to find and correlate matching time information. The system iterates through each record in the event data table, each record containing an entity identifier (EID), an event code (event_code), and a sequence index (index).
[0063] The specific steps are as follows: Calculation and lookup: For each event record, the same hash function and hash table as in S301 are used to calculate the hash value based on its own (EID, index) composite primary key and locate the corresponding bucket in the hash table.
[0064] Key-value matching: Within this bucket (considering possible conflicts), perform precise key-value comparisons (i.e., compare whether EID and index are exactly equal) to find matching time records.
[0065] Record association: If a matching time record is found, the time value (date) in that time record is logically or physically associated with the event code (event_code) in the current event record to form a matching pair. If no match is found, the event record may be marked as having no matching time or processed accordingly based on business requirements.
[0066] S303. Result Generation: This step assembles the successfully matched pairs into the final output. For each pair of time records and event records that successfully match in S302, the system extracts and merges key fields to generate a reconstructed medical data record. This record contains at least: Entity ID (EID), sequence index, date value (from the time record), and event code (from the event record).
[0067] These records are output sequentially or written to a new result table, thus completing the entire process of accurately aligning and reconstructing the separately stored time and event information based on an explicit sequence index.
[0068] Variations and optimized implementation methods: Parallel Probing: In S302, the probe table (event data table) can be divided into multiple shards, and multiple threads or processes can perform hash lookup and matching operations in parallel, which greatly improves the association speed.
[0069] Memory management: If the time data table is too large to be loaded into memory at once, a strategy of building and probing in blocks can be adopted, or a disk-based hybrid hash join algorithm can be used.
[0070] Distributed joins: In a clustered environment, if both tables are extremely large, a sharded hash join (ShuffledHashJoin) can be used. This involves first hashing both the time schedule and event tables according to their EIDs and then shuffling them to the same compute nodes. Then, the in-memory hash join process described above is executed independently on each node.
[0071] Join algorithm selection: As an alternative, if the input data has been sorted by the join key or optimized by index, the system can also automatically select Sort-MergeJoin as the execution strategy to achieve better performance.
[0072] In some embodiments, the index-aligned medical data reconstruction system may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the index-aligned medical data reconstruction system may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality for medical data reconstruction based on index alignment.
[0073] In this embodiment, the index-aligned medical data reconstruction system can be divided into multiple functional modules according to its functions, such as... Figure 2 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.
[0074] The field splitting module is used to convert multiple time field values corresponding to the same entity identifier into a time data table containing the entity identifier, time value, and time sequence identifier for the source data table; at the same time, it splits the field values containing multiple event codes corresponding to the same entity identifier into an event data table containing the entity identifier and a single event code. The index building module is used to directly use the time sequence identifiers in the time data table as sequence indexes; and to generate a continuous sequence index for each event record based on the original order of the event codes under each entity identifier in the event data table. The data reconstruction module is used to associate records with the same entity identifier and the same sequence index value in the time data table and the event data table to obtain the reconstructed medical data.
[0075] Figure 3 The index-aligned medical data reconstruction method provided in this application embodiment can be applied to devices. Those skilled in the art will understand that the device structures involved in the embodiments of this invention do not constitute a limitation on the device; a device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.
[0076] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0077] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is able to perform some or all of the steps in the above method embodiments.
[0078] The processor 310 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.
[0079] The communication unit 330 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.
[0080] The present invention also provides a computer medium, wherein the computer medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0081] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.
[0082] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.
[0083] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.
[0084] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0085] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.
[0086] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.
Claims
1. A method for reconstructing medical data based on index alignment, characterized in that, include: For the source data table, multiple time field values corresponding to the same entity identifier are converted into a time data table containing the entity identifier, time value, and time sequence identifier; at the same time, field values containing multiple event codes corresponding to the same entity identifier are split into event data tables containing the entity identifier and a single event code. For the time data table, the time sequence identifier is directly used as the sequence index; for the event data table, a continuous sequence index is generated for each event record based on the original order of the event codes under each entity identifier. By associating records with the same entity identifier and the same sequence index value in the time data table and the event data table, the reconstructed medical data is obtained.
2. The method according to claim 1, characterized in that, Multiple time field values corresponding to the same entity identifier are converted into a time data table containing the entity identifier, time value, and time sequence identifier, including: Perform a reverse pivot operation on multiple time fields representing the same time dimension in the source data table, convert the field name of each time field into the time sequence identifier, and use the corresponding field value as the time value to generate the time data table.
3. The method according to claim 2, characterized in that, Convert the field name of each time field to the time sequence identifier, including: Extract the numeric suffix from the field name and use the numeric suffix as the time sequence identifier for the corresponding record.
4. The method according to claim 1, characterized in that, The field values containing multiple event codes corresponding to the same entity identifier are split into an event data table containing the entity identifier and a single event code, including: The field value is split into multiple independent event codes based on a preset delimiter; Each extracted event code is associated with its corresponding entity identifier to form an event record in the event data table.
5. The method according to claim 1, characterized in that, Based on the original order of event codes under each entity identifier, a continuous sequence index is generated for each event record, including: In the event data table, for each entity identifier group, a window function is applied to generate consecutive serial numbers, and these serial numbers are used as sequence indexes; The window function is the ROW_NUMBER() function, which groups entities by entity identifier using the PARTITIONBY clause and performs stable sorting based on the order of event codes in the original field values using the ORDERBY clause.
6. The method according to claim 1, characterized in that, For the event data table, based on the original order of event codes under each entity identifier, a continuous sequence index is generated for each event record, including: The event records under each entity identifier group are distributed to multiple parallel processing units; wherein, each event record in the event data table corresponds to one entity identifier and one event code obtained after splitting. Within each of the parallel processing units, a local sequential number starting from zero is generated for the event records assigned to it; The global starting offset of the record processed by each of the parallel processing units is determined through a parallel, hierarchical aggregation computation. The local consecutive number of each event record is added to the global starting offset of its corresponding parallel processing unit to obtain the final consecutive sequence index of the event record.
7. The method according to claim 1, characterized in that, By associating records with the same entity identifier and the same sequence index value in the time data table and the event data table, the reconstructed medical data is obtained, including: Using the time data table as the construction table, based on the composite primary key composed of the entity identifier and the sequence index, the hash value is calculated by a hash function, and the corresponding record is stored in the corresponding position of the memory hash table; Using the event data table as a probe table, traverse each record, calculate the hash value based on the same composite primary key, and search for the time record with the same composite primary key value in the memory hash table. Then, associate the found time record with the event record. Merge each successfully associated time record and event record, and output a reconstructed record containing entity identifier, time value, event code and sequence index.
8. A medical data reconstruction system based on index alignment, characterized in that, include: The field splitting module is used to convert multiple time field values corresponding to the same entity identifier into a time data table containing the entity identifier, time value, and time sequence identifier for the source data table; at the same time, it splits the field values containing multiple event codes corresponding to the same entity identifier into an event data table containing the entity identifier and a single event code. The index building module is used to directly use the time sequence identifiers in the time data table as sequence indexes; and to generate a continuous sequence index for each event record based on the original order of the event codes under each entity identifier in the event data table. The data reconstruction module is used to associate records with the same entity identifier and the same sequence index value in the time data table and the event data table to obtain the reconstructed medical data.
9. A medical data reconstruction device based on index alignment, characterized in that, include: Memory for storing index-aligned medical data reconstruction procedures; A processor, configured to implement the steps of the index-aligned medical data reconstruction method as described in any one of claims 1-7 when executing the index-aligned medical data reconstruction procedure.
10. A computer-readable medium storing a computer program, characterized in that, The readable medium stores an index-aligned medical data reconstruction program, which, when executed by a processor, implements the steps of the index-aligned medical data reconstruction method as described in any one of claims 1-7.