Medical big data file exporting method and system based on memory mapping file and terminal

Abstract

The application provides a medical big data file export method, system and terminal based on a memory mapping file, wherein data in a processing process is written into a memory mapping file by using a memory mapping file technology, memory occupation is reduced, server lag is reduced, and throughput is improved. 2 In a data merging process, a scheme of first sorting and then traversing by using a pointer is adopted, the complexity of an algorithm is reduced from O(n) to O(n), export time consumption is greatly shortened, server load can be greatly reduced in cooperation with memory occupation reduction, and a large amount of data can be stably exported in a short time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a medical big data file export method and system based on a memory mapping file and a terminal. BACKGROUND

[0002] When conducting scientific research, hospitals often need to import the diagnosis and treatment data of patients in the hospital into a third-party statistical analysis system for analysis. According to different analysis algorithms, the third-party statistical analysis system may need a large amount of patient data and may need to be spliced into one patient row or one visit row. The format may require Csv or Excel, Spss, Sas. Due to the limitation of the hardware resources of the hospital server, the conventional export method may cause server lag and memory overflow when splicing a large amount of patient data, or may not generate a file for a long time, thereby causing export failure or long-time failure to export successfully. SUMMARY

[0003] In view of the above-mentioned shortcomings of the prior art, the present application aims to provide a medical big data file export method and system based on a memory mapping file and a terminal, which can solve the technical problems of server lag and memory overflow when splicing a large amount of patient data, or long-time failure to generate a file, thereby causing export failure or long-time failure to export successfully due to the limitation of the hardware resources of the hospital server in the prior art.

[0004] To achieve the above-mentioned purposes and other related purposes, the present application provides a medical big data file export method based on a memory mapping file, which comprises the following steps: selecting the required variables; grouping and querying the variable data according to the domains, and writing the domain data of each variable domain obtained by the query into the corresponding generated memory mapping file; performing row merging on the domain data of each variable domain according to the target variable dimension to obtain variable domain row merging data of the target variable dimension, and writing the variable domain row merging data into the generated memory mapping file of the target variable dimension; and generating a final file by reading the data from the memory mapping file of the target variable dimension row by row and exporting the final file.

[0005] In an embodiment of the present application, the grouping and querying of the variable data according to the domains, and the writing of the domain data of each variable domain obtained by the query into the corresponding generated memory mapping file comprises: grouping and querying the variable data according to the domains, and sorting according to the patient number and the visit number to obtain the domain data of each variable domain; and sequentially performing reading and writing operations on the domain data of each variable domain to write the domain data of each variable domain into the corresponding generated memory mapping file.

[0006] In an embodiment of the present application, the domain data of the current variable domain is read row by row and cell by cell, and the cell statistical information of each cell obtained by the statistics is recorded into the row information memory object of the current variable domain; wherein the cell statistical information includes: cell data volume, offset, patient number and visit number; a memory mapping file corresponding to the file size is created according to the cell statistical information of the current variable domain, and the cell data of the current variable domain is written into the memory mapping file, and the file information of the memory mapping file is recorded into the file information memory object of the current variable domain.

[0007] In an embodiment of the present application, the row-merging data of the target variable dimension of each variable domain is obtained by performing row-merging according to the target variable dimension, and writing into the generated memory mapping file of the target variable dimension includes: performing intra-domain data assembly operation on the domain data of each variable domain according to the target variable dimension, so as to write the obtained target variable dimension assembly data of each variable domain into the corresponding generated assembly memory mapping file; performing all-domain row data assembly operation based on the assembly memory mapping file of each variable domain, obtaining the variable domain row-merging data of the target variable dimension, and writing the variable domain row-merging data of the target variable dimension into the corresponding generated memory mapping file of the target variable dimension; wherein the type of the target variable dimension includes: patient dimension and visit dimension; when the target variable dimension is the patient dimension, one target variable object is one patient; when the target variable dimension is the visit dimension, one target variable object is one visit.

[0008] In an embodiment of the present application, the intra-domain data assembly operation includes: traversing all cell statistical information under each target variable object in the row information memory object of the current variable domain, and taking all cell data under each target variable object of the current variable domain as a row to obtain intra-domain assembly row data of each target variable object of the current variable domain, and recording all cell statistical information in each intra-domain assembly row data into the corresponding merging row information memory object; after the intra-domain assembly row data of all target variable objects of the current variable domain is obtained, an assembly memory mapping file corresponding to the file size is created according to all cell statistical information of each target variable object of the current variable domain, and the obtained intra-domain assembly row data of each target variable object is written into the assembly memory mapping file as the target variable dimension assembly data of the current variable domain.

[0009] In an embodiment of the present application, the all-domain row data assembly operation includes: obtaining target variable numbers of all variable domains and sorting after deduplication; traversing the merged row information memory object of each variable domain according to the sorted target variable number pointers, comparing the target variable number of the traversed cell statistical information with the current target variable number, and updating the table information when they are the same; after the traversal is completed, the data of all variable domains of each target variable object is spliced into a row to obtain all-domain row splicing data of each target variable object according to the sorted target variable numbers, and the all-domain row splicing data of each target variable object is sequentially written into the corresponding generated memory mapping file of the target variable dimension until all the all-domain row splicing data of the target variable object is written.

[0010] In an embodiment of the present application, if the default export is selected, after the domain data of each variable domain of the query is written into the corresponding generated memory mapping file, the data is read row by row from each variable domain memory mapping file to generate a final file and export.

[0011] In an embodiment of the present application, the variable domain includes: a patient variable domain, a visit variable domain, a diagnosis variable domain, a test variable domain, a surgery variable domain, and an electrocardiogram variable domain.

[0012] To achieve the above object and other related objects, the present application provides a medical big data file export system based on a memory mapping file, which includes: a variable selection module for selecting the required variables; a variable domain data query module connected to the variable selection module for grouping and querying the variable data by domain, and writing the domain data of each variable domain of the query into the corresponding generated memory mapping file; a data merging module connected to the variable domain data query module for performing row merging on the domain data of each variable domain according to the target variable dimension to obtain variable domain row merging data of the target variable dimension, and writing into the generated memory mapping file of the target variable dimension; and a file export module connected to the data merging module for reading data row by row from the memory mapping file of the target variable dimension to generate a final file and export.

[0013] To achieve the above object and other related objects, the present application provides an electronic terminal, which includes: one or more memories and one or more processors; the one or more memories are used for storing computer programs; the one or more processors are connected to the memories and are used for running the computer programs to execute the medical big data file export method based on a memory mapping file.

[0014] As described above, the present application is a medical big data file export method, system and terminal based on memory mapping files, which has the following beneficial effects: the present application uses memory mapping file technology to write data in the processing process into a memory mapping file, reduces memory occupation, reduces server lag, and improves throughput. In the data merging process, the scheme of sorting first and then using a pointer to traverse is adopted, which reduces the complexity of the algorithm from O(n 2 ) to O(n), greatly shortens the export time consumption, and cooperates with the reduction of memory occupation, which can greatly reduce the server load, and can make a large amount of data be exported in a short time. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The flowchart shown is a flowchart of the medical big data file export method based on memory mapping files in an embodiment of the present application.

[0016] Figure 2 The front-end variable selection interface shown is a schematic diagram of an embodiment of the present application.

[0017] Figure 3 The flowchart shown is a flowchart of the medical big data file export method based on memory mapping files in an embodiment of the present application.

[0018] Figure 4 The structural diagram shown is a structural diagram of the medical big data file export system based on memory mapping files in an embodiment of the present application.

[0019] Figure 5 The structural diagram shown is a structural diagram of the electronic terminal in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the present application will be described below through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present specification. The present application can also be implemented or applied through other different specific embodiments, and each detail in the present specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, unless the above context clearly dictates otherwise. Spatially relative terms, such as "upper," "lower," "left," "right," "bottom," "top," "horizontal," "vertical," and the like, can be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be further understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientations depicted in the figures. For example, if a device described is turned over in use, a relative physical property previously described as on the bottom surface can now be

[0022] Throughout this specification, when it is said that a certain part is "connected" to another part, it includes not only the case of "direct connection" but also the case of "indirect connection" in which other elements are interposed therebetween. In addition, when it is said that a certain part "includes" a certain constituent element, unless it is specifically stated to the contrary, other constituent elements are not excluded but it means that other constituent elements can be further included.

[0023] The terms first, second, third, etc. that are mentioned herein are used to explain various parts, components, regions, layers and / or sections, but are not limited thereto. These terms are used only to distinguish a certain part, component, region, layer or section from other parts, components, regions, layers or sections. Therefore, the first part, component, region, layer or section described below can be referred to as the second part, component, region, layer or section within the scope of the present application.

[0024] Further, as used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and / or "including," when used herein, specify the presence of stated features, operations, elements, components, items, and / or objects, but do not preclude the presence or addition of one or more other features, operations, elements, components, items, and / or objects. As used herein, the term "or" and "and / or" is construed to be inclusive, or to mean "either" or "any combination of the items." Thus, "A, B, or C" or "A, B, and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B, and C." This definition also applies to other common terms such as "at least one of," "one, but not both," and "one, or the other, but not both."

[0025] The application provides a medical big data file export method based on a memory mapping file. The data in a processing process is written into a memory mapping file by using a memory mapping file technology, so that the memory occupation is reduced, the server lag is reduced, and the throughput is improved. In the data merging process, the scheme of sorting first and then traversing by using a pointer is adopted, so that the complexity of the algorithm is reduced from O(n 2 ) to O(n), the export time consumption is greatly shortened, and in combination with the reduction of the memory occupation, the server load can be greatly reduced, and a large amount of data can be stably exported in a short time.

[0026] The embodiments of the application are described in detail below with reference to the accompanying drawings, so that the person skilled in the art of the application can easily implement the application. The application can be embodied in various different forms, and is not limited to the embodiments described herein.

[0027] As Figure 1 A flowchart of a medical big data file export method based on a memory mapping file in an embodiment of the application is shown.

[0028] The method comprises the following steps.

[0029] Step S1: selecting required variables.

[0030] In an embodiment, a researcher can select exported variables from the front end to determine the data range. The variables are grouped by domains, including patient variable domains, clinic variable domains, diagnosis variable domains, test variable domains, operation variable domains and electrocardiogram variable domains. Each variable domain corresponds to a plurality of variables. For example, as shown in the table, the variables are grouped by domains, including patient basic information, clinic information, diagnosis, test, operation, electrocardiogram and the like. The variables of the patient basic information include patient number, name, nationality, ethnicity, date of birth, telephone number, certificate and the like. Figure 2

[0031] Step S2: grouping and querying the variable data by domains, and writing the domain data of each variable domain obtained by the querying into a corresponding generated memory mapping file.

[0032] In an embodiment, step S2 comprises:

[0033] The variable data is grouped and queried by domains, and is sorted according to the patient number and the clinic number to obtain the domain data of each variable domain. Specifically, the data of each variable is sequentially queried according to the variable domain, and the data of each variable domain is obtained by sorting according to the patient number and the clinic number during the querying. The domain data of each variable domain includes the data of the variables belonging to the variable domain and sorted according to the patient number and the clinic number.

[0034] ​The reading and writing operations are sequentially performed on the domain data of each variable domain to write the domain data of each variable domain into a corresponding generated memory mapping file.

[0035] In an embodiment, the reading and writing operations include:

[0036] The domain data of the current variable domain is read row by row and cell by cell, and the cell statistical information of each cell is recorded into a row information memory object of the current variable domain, and the data of each cell is written into another data object; wherein the cell statistical information includes: cell data volume (data length and size), offset (position information of the cell data), patient number, and visit number;

[0037] A memory mapping file with a corresponding file size is created according to the cell statistical information of the current variable domain, and the cell data of the current variable domain is written into the memory mapping file using a disk sequential writing method, and the file information of the memory mapping file, such as file name and size, is recorded into a file information memory object of the current variable domain.

[0038] Step S3: The domain data of each variable domain is row-merged according to the target variable dimension to obtain variable domain row-merged data of the target variable dimension, and is written into a generated memory mapping file of the target variable dimension.

[0039] In an embodiment, step S3 includes:

[0040] The domain data of each variable domain is in-domain data assembled according to the target variable dimension to write the obtained target variable dimension assembled data of each variable domain into a corresponding generated assembled memory mapping file;

[0041] The all-domain row data assembly operation is performed based on the assembled memory mapping files of the variable domains to obtain variable domain row-merged data of the target variable dimension, and the variable domain row-merged data of the target variable dimension is written into a corresponding generated memory mapping file of the target variable dimension;

[0042] The type of the target variable dimension includes: patient dimension and visit dimension; when the target variable dimension is the patient dimension, one target variable object is one patient; when the target variable dimension is the visit dimension, one target variable object is one visit.

[0043] In an embodiment, the in-domain data assembly operation includes:

[0044] The total number of cells recorded under the target variable object is counted according to the row information memory object of the previous variable domain, and the number of columns of the current variable domain table is generated according to this;

[0045] traverse all cell statistical information under each target variable object in the current variable domain's row information memory object, calculate the record position of all cell data under each target variable object in the current variable domain, and take out all cell data under each target variable object in the current variable domain to assemble a row, obtain the in-domain assembled row data of each target variable object in the current variable domain, and record all cell statistical information in each in-domain assembled row data to the corresponding merged row information memory object;

[0046] After the in-domain assembled row data of all target variable objects in the current variable domain are obtained, an assembled memory mapping file with a corresponding file size is created according to all cell statistical information of each target variable object in the current variable domain and the column number of the current variable domain table, and the obtained in-domain assembled row data of each target variable object is sequentially written as target variable dimension assembly data of the current variable domain according to row in the assembled memory mapping file, and the file information of the memory mapping file is recorded to the file information memory object of the current variable domain.

[0047] In an embodiment, the total domain row data assembly operation includes:

[0048] According to the merged row information memory object of each variable domain, the maximum column number in each variable domain is counted, and the table column number is generated according to this;

[0049] The target variable numbers of all variable domains are obtained, and after deduplication, they are sorted;

[0050] According to the sorted target variable number pointers, the merged row information memory objects of the variable domains are traversed, the target variable numbers of the traversed cell statistical information are compared with the current target variable number, and the table information is updated when they are the same. Specifically, according to the sorted target variable numbers, the merged row information memory objects of the variable domains are traversed, and a counter is specified for each variable domain. Since they are all sorted by patient number, the patient row information object can be directly obtained according to the index and the subscript, and compared with the currently traversed patient. If they are the same, the number is taken, the counter is incremented by 1, and if they are not the same, it is skipped. Compared with nested loop traversal, the algorithm complexity can be reduced to O(n).

[0051] After the traversal is completed, all variable domain data of each target variable object is spliced into a row to obtain total domain row splicing data of each target variable object according to the sorted target variable numbers, a memory mapping file of the target variable dimension is generated according to the statistical information of the merged row information memory object of each variable domain, the generated table column number, and the counter of each variable domain, and the total domain row splicing data of each target variable object is sequentially written into the corresponding generated memory mapping file of the target variable dimension until all total domain row splicing data of all target variable objects are written.

[0052] Step S4: generating a final file from the memory mapping file of the target variable dimension and exporting.

[0053] In an embodiment, according to the merged row information memory object of each variable domain, data is read from the memory mapping file of the target variable dimension row by row and written into a file of a corresponding format (an Excel or Csv file), and finally a final file is generated and exported.

[0054] For example, according to the merged row information memory object of each variable domain, patient data is read row by row and written into an Excel file stream, and finally an Excel file is generated.

[0055] In an embodiment, after step 2 is performed, it can also be judged whether to select default export, if the default export is selected, data is read from the memory mapping file of each variable domain row by row to generate a final file and export. If the default export is not selected, step 3 needs to be performed.

[0056] In order to better describe the medical big data file export method based on the memory mapping file, the following specific embodiments will be described.

[0057] Embodiment 1: a medical big data file export method based on a memory mapping file. Figure 3 The flowchart of the medical big data file export method based on the memory mapping file of the present embodiment.

[0058] The research personnel can select the exported variables by the front end to determine the data range, and each domain will generate an Excel worksheet (if it is Csv, it is embodied by multiple files) when the default export is performed. When a patient row is selected, the data will be merged into a worksheet according to the patient dimension and spliced into a row, and the data of multiple patient dimensions will be displayed by adding a column. One visit per row will merge the data into a worksheet according to the visit dimension and splice it into a row, and the data of multiple visit dimensions will be displayed by adding a column. The export process is divided into a data query stage, a data processing stage, and a file generation stage.

[0059] The method comprises:

[0060] Step 1: selecting the exported variables by the front end.

[0061] Step 2: grouping and querying the data according to the domains, comprising the following steps:

[0062] Step 21: starting to query the data according to the domains in turn, and sorting according to the patient number and then the visit number when querying;

[0063] Step 22: Read the queried data row by row and cell by cell, count the data volume of each cell, the patient offset, and the current row patient number and visit number, and record them to the row information memory object, and then write the cell content to another data object.

[0064] Step 3: Write the field data of each variable field in the query to the corresponding generated memory mapping file.

[0065] After all the cells in the current field are read, the file size is calculated according to the statistical information to create a memory mapping file, and then the data object is written to the file using the sequential writing method, and the file information (file name, size) is recorded to the file information memory object.

[0066] Steps S2 and S3 are performed in the data query phase, and after all the variable fields are successfully queried, the next phase is entered, and steps 2 and 3 are repeated until all the variable fields are processed. When each field query is completed, the data is written to the memory mapping file to save memory.

[0067] Step S3: Determine whether to export by default.

[0068] If the default export is selected, step S6 is executed; if the non-default export is selected, step S4 is executed.

[0069] Step S4: Convert one patient row / one visit row in the variable field to write to a new memory mapping file.

[0070] If one patient row is converted, the specific steps are as follows:

[0071] Step 411: According to the row information memory object, the total number of patient records is counted, and columns and table headers are generated according to this;

[0072] Step 412: According to the row information object, the patient dimension is traversed, the positions of all patient data records are calculated, all cell data of the current patient are taken out, assembled into a row, and then written to a new data object, and the data volume, offset, and current row patient number and visit number of each cell are recorded to the row information memory object.

[0073] Step 413: When all patients are processed, the file size is calculated according to the row information memory object to create a memory mapping file, and then the data object is written to the file using the sequential writing method, and the file information is recorded to the file information memory object.

[0074] Repeat steps 411, 412, and 413 until all patients are processed, and then execute subsequent steps 511 to 513.

[0075] If one visit row is converted, the specific steps are as follows:

[0076] Step 421: According to the row information memory object statistics, the total number of records is recorded, and the column and table header are generated according to this;

[0077] Step 422: According to the row information object traversal visit dimension, the total data record position of each visit is calculated, the current visit all cell data is taken out, assembled to a line and re-written to a new data object, and the data volume, offset and current row patient number visit number record of each cell are recorded to the row information memory object.

[0078] Step 423: When all visits are processed, the file size is calculated according to the row information memory object, and then the data object is written in the order of writing to the file, and the file information is recorded to the file information memory object.

[0079] Repeat steps 421, 422, and 423 until all visits are processed, and then execute subsequent step 521.

[0080] Step 5: Merge all variable fields into one patient line / one visit line, and write into a new memory mapping file.

[0081] If one patient line is converted, the specific steps are as follows:

[0082] Step 511: According to the row information memory object, the maximum number of columns in each field is counted, and the column and table header are generated according to this.

[0083] Step 512: Get the sorted patient number after removing the duplicate patient number in the front part of all fields.

[0084] Step 513: Traverse the patient number, and then traverse the row information memory object of each field generated in step 412 according to the current patient number. Set a counter for each variable field, and directly compare the current traversed patient by using the sorted patient number through indexing. If they are the same, extract the data and count the counter by 1. If they are different, skip this step. This method avoids complex nested loops and optimizes the algorithm complexity to O(n). Subsequently, according to the comparison result, the patient data across the fields is spliced into a line, the row information is counted and recorded, and then the memory mapping file is written in sequence. Repeat this process until all patient data is processed, and finally count and record the detailed information of the entire memory mapping file.

[0085] If one visit line is converted, the specific steps are as follows:

[0086] Step 521: According to the row information memory object, the maximum number of columns in each field is counted, and the column and table header are generated according to this.

[0087] Step 522: Get the sorted visit number after removing the duplicate visit number in the front part of all fields.

[0088] Step 523: traverse the visit number, and then traverse the row information memory object of each domain generated in step 422 according to the current visit number, set a counter for each variable domain, directly compare the current traversed visit number by index using the sorted visit number, extract the data and count the counter by 1 if the same, and skip if different. This method avoids complex nested loops and optimizes the algorithm complexity to O(n). Subsequently, the cross-domain visit data is spliced into a row according to the comparison result, the row information is counted and recorded, and then sequentially written into the memory mapping file. Repeat this process until all visit data is processed, and finally count and record the detailed information of the entire memory mapping file.

[0089] Step 6: read data from the memory mapping file and write it into the final file.

[0090] Similar to the principle of the above embodiment, the present application provides a medical big data file export system based on a memory mapping file.

[0091] The following provides specific embodiments in combination with the drawings:

[0092] As Figure 4 The structure of a medical big data file export system based on a memory mapping file in an embodiment of the present application is shown.

[0093] The system comprises:

[0094] A variable selection module 1 for selecting the required variables;

[0095] A variable domain data query module 2 connected to the variable selection module 1 for grouping and querying the variable data by domain, and writing the domain data of each queried variable domain into the corresponding generated memory mapping file;

[0096] A data merging module 3 connected to the variable domain data query module 2 for performing row merging on the domain data of each variable domain according to the target variable dimension to obtain variable domain row merging data of the target variable dimension, and writing it into the generated memory mapping file of the target variable dimension;

[0097] A file export module 4 connected to the data merging module 3 for reading data from the memory mapping file of the target variable dimension row by row to generate a final file and export it.

[0098] It should be noted that Figure 4 The division of each module in the system embodiment is only a logical functional division, and all or part of them can be integrated into one physical entity, or physically separated. And these units can all be implemented in the form of software called by a processing element; or all can be implemented in the form of hardware; or part of the units can be implemented in the form of software called by a processing element, and part of the units can be implemented in the form of hardware.

[0099] Since the implementation principle of the medical big data file export system based on the memory mapping file has been described in the foregoing embodiments, it is not repeated here.

[0100] The medical big data file export method based on the memory mapping file provided by the embodiment of the application can be implemented on the terminal side or the server side. As for the hardware structure of the electronic terminal, please refer to Figure 5 , an optional hardware structure schematic diagram of the electronic terminal 1000 provided by the embodiment of the application. The terminal 1000 can be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The terminal 1000 includes at least one processor 1001, a memory 1002, at least one network interface 10010 and a user interface 1009. Various components in the apparatus are coupled together through a bus system 1005. It can be understood that the bus system 1005 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 1005 also includes a power bus, a control bus and a status signal bus. However, in order to clearly illustrate, all kinds of buses are marked as the bus system in Figure 5 .

[0101] The user interface 1009 can include a display, a keyboard, a mouse, a trackball, a click gun, a key, a button, a touch pad or a touch screen, etc.

[0102] It can be understood that the memory 1002 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM, Static Random Access Memory), synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory). The memory described in the embodiment of the application is intended to include but not limited to these and any other suitable category of memory.

[0103] The memory 1002 in the embodiments of the present application is configured to store various types of data to support the operation of the terminal 1000. Examples of the data include any executable programs for operating on the terminal 1000, such as an operating system 10021 and an application program 10022. The operating system 10021 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application program 10022 can include various application programs, such as a media player (MediaPlayer), a browser (Browser), and the like, for implementing various application services. The application program 10022 can include the method for exporting a medical big data file based on a memory mapping file according to the embodiments of the present application.

[0104] The method according to the embodiments of the present application can be applied to the processor 1001 or implemented by the processor 1001. The processor 1001 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 1001. The processor 1001 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 1001 can implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application. The general-purpose processor 1001 can be a microprocessor or any conventional processor, etc. The steps of the method for optimizing accessories according to the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium, which is located in the memory. The processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0105] In the exemplary embodiments, the terminal 1000 can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), or the like, for executing the above method.

[0106] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by computer program related hardware. The aforementioned computer program can be stored in a computer readable storage medium. The program executes the steps of the above-mentioned method embodiments when executed; and the aforementioned storage medium includes ROM, RAM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device, flash memory, U disk, mobile hard disk or any other medium capable of storing desired program code in the form of instructions or data structure and being accessed by a computer. In addition, any connection can be appropriately referred to as a computer readable medium. For example, if instructions are sent from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave is included in the definition of the medium. However, it should be understood that the computer readable storage medium and the data storage medium do not include connections, carriers, signals or other temporary media, but are intended for non-transitory, tangible storage media. As used in the application, magnetic disks and optical disks include compact disks (CD), laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, wherein magnetic disks usually magnetically copy data, and optical disks use lasers to optically copy data.

[0107] In the embodiments provided in the present application, the computer readable storage medium can include read-only memory, random access memory, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device, flash memory, U disk, mobile hard disk, or any other medium capable of storing desired program code in the form of instructions or data structure and being accessed by a computer. In addition, any connection can be appropriately referred to as a computer readable medium. For example, if instructions are sent from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave is included in the definition of the medium. However, it should be understood that the computer readable storage medium and the data storage medium do not include connections, carriers, signals or other temporary media, but are intended for non-transitory, tangible storage media. As used in the application, magnetic disks and optical disks include compact disks (CD), laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, wherein magnetic disks usually magnetically copy data, and optical disks use lasers to optically copy data.

[0108] Compared with the prior art, the present application has the following specific advantages:

[0109] 1. By introducing the memory mapping file technology, the present application can directly write intermediate results into files when processing large-scale data, effectively reducing the memory occupation during program running. This innovation not only relieves the memory pressure of the server, reduces the phenomenon of lag caused by insufficient memory, but also significantly improves the data throughput of the system, ensuring efficient and stable performance.

[0110] 2. In the data merging stage, the present application abandons the traditional nested loop traversal method, and instead adopts an efficient strategy of first sorting the data and then using a pointer for linear traversal. This algorithm innovation reduces the algorithm complexity from O(n 2 ) to O(n), greatly shortening the data export time and significantly reducing the server resource occupation time, especially when processing massive data, this advantage is more prominent, bringing a more smooth data processing experience to users.

[0111] In summary, the medical big data file export method, system and terminal based on the memory mapping file of the present application, by using the memory mapping file technology, the data in the processing process is written into the memory mapping file, reduces the memory occupation, reduces the server lag, improves the throughput. In the data merging process, the scheme of sorting first and then using pointer traversal is adopted, the complexity of the algorithm is reduced from O(n 2 ) to O(n), which greatly shortens the export time consumption, and cooperates with the reduction of memory occupation, can greatly reduce the server load, and can make a large amount of data be exported in a short time. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has high industrial utilization value.

[0112] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought disclosed by the present application should be covered by the claims of the present application.

Claims

1. A method for exporting medical big data files based on memory-mapped files, characterized in that, The method includes: Select the desired variables; The variable data is queried in groups by domain, and the domain data of each queried variable domain is written to the corresponding generated memory-mapped file; The process involves merging the domain data of each variable domain according to the target variable dimension to obtain the target variable dimension's row-merged data, and then writing it into the generated target variable dimension's memory-mapped file. This process includes: performing intra-domain data assembly operations on the domain data of each variable domain according to the target variable dimension, writing the obtained target variable dimension assembled data for each variable domain into the corresponding generated assembled memory-mapped file; performing row data assembly operations on all domains based on the assembled memory-mapped files of each variable domain to obtain the target variable dimension's row-merged data, and writing the target variable dimension's row-merged data into the corresponding generated target variable dimension's memory-mapped file; where the target variable dimension types include: patient dimension and visit dimension; when the target variable dimension is patient dimension, one target variable object represents one patient; when the target variable dimension is visit dimension, one target variable object represents one visit. Data is read line by line from the memory-mapped file representing the target variable dimension to generate the final file and then exported.

2. The method for exporting medical big data files based on memory-mapped files according to claim 1, characterized in that, The step of grouping and querying variable data by domain, and writing the domain data of each queried variable domain into the corresponding generated memory-mapped file includes: The variable data is grouped and queried by domain, and sorted according to patient number and visit number to obtain the domain data for each variable domain. For each variable field, read and write operations are performed sequentially to write the field data of each variable field to the corresponding generated memory-mapped file.

3. The method for exporting medical big data files based on memory-mapped files according to claim 2, characterized in that, The read and write operations include: The domain data of the current variable domain is read row by row and cell by cell, and the statistical information of each cell is recorded in the row information memory object of the current variable domain; wherein, the cell statistical information includes: cell data volume, offset, patient number and visit number; Create a memory-mapped file of the corresponding file size based on the statistical information of each cell in the current variable field, write the data of each cell in the current variable field into the memory-mapped file, and record the file information of the memory-mapped file into the file information memory object of the current variable field.

4. The method for exporting medical big data files based on memory-mapped files according to claim 1, characterized in that, The intra-domain data assembly operation includes: In the current variable domain's row information memory object, iterate through all cell statistics under each target variable object, retrieve all cell data under each target variable object in the current variable domain and assemble them into a row, obtain the domain-in-domain assembled row data of each target variable object in the current variable domain, and record all cell statistics in each domain-in-domain assembled row data into the corresponding merged row information memory object; After all the domain-in-domain assembly row data of all target variable objects in the current variable domain has been obtained, an assembly memory-mapped file of the corresponding file size is created based on the statistical information of all cells of each target variable object in the current variable domain. The domain-in-domain assembly row data of each target variable object is then written into the assembly memory-mapped file as the target variable dimension assembly data of the current variable domain.

5. The method for exporting medical big data files based on memory-mapped files according to claim 1, characterized in that, The assembly operation of all field row data includes: Obtain the target variable numbers for all variable domains, and sort them after removing duplicates; Based on the sorted pointers to the target variable numbers, the table information memory objects of the merged rows of each variable field are traversed. The target variable numbers of the statistical information of the traversed cells are compared with the current target variable numbers, and the table information is updated when they are the same. After the traversal is completed, according to the sorted target variable numbers, the data of all variable fields of each target variable object are concatenated into a row to obtain the concatenated data of all fields of each target variable object. The concatenated data of all fields of each target variable object is then written sequentially into the memory-mapped file of the corresponding target variable dimension until the concatenated data of all fields of all target variable objects is written.

6. The method for exporting medical big data files based on memory-mapped files according to claim 1, characterized in that, If the default export is selected, after writing the domain data of each queried variable field to the corresponding generated memory-mapped file, the data is directly read line by line from the memory-mapped file of each variable field to generate the final file and then exported.

7. The method for exporting medical big data files based on memory-mapped files according to claim 1, characterized in that, The variable domains include: patient variable domain, medical visit variable domain, diagnosis variable domain, laboratory test variable domain, surgical variable domain, and electrocardiogram variable domain.

8. A medical big data file export system based on memory-mapped files, characterized in that, The system includes: The variable selection module is used to select the desired variable; The variable domain data query module, connected to the variable selection module, is used to query variable data by domain and write the domain data of each queried variable domain into the corresponding generated memory-mapped file. The data merging module, connected to the variable domain data query module, is used to perform row merging on the domain data of each variable domain according to the target variable dimension to obtain the variable domain row merged data of the target variable dimension, and write it to the generated memory-mapped file of the target variable dimension. The process includes: performing intra-domain data assembly operations on the domain data of each variable domain according to the target variable dimension, so that the obtained target variable dimension assembled data of each variable domain is written to the corresponding generated assembled memory-mapped file; performing all domain row data assembly operations based on the assembled memory-mapped files of each variable domain to obtain the variable domain row merged data of the target variable dimension, and writing the variable domain row merged data of the target variable dimension to the corresponding generated memory-mapped file of the target variable dimension; wherein, the type of the target variable dimension includes: patient dimension and visit dimension; when the target variable dimension is patient dimension, one target variable object represents one patient; when the target variable dimension is visit dimension, one target variable object represents one visit. The file export module, connected to the data merging module, is used to read data line by line from the memory-mapped file of the target variable dimension to generate the final file and export it.

9. An electronic terminal, characterized in that, include: One or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors are connected to the memory and are used to run the computer program to perform the method as described in any one of claims 1 to 7.

Images