Medical big data file exporting method and system based on memory mapping file and terminal

Abstract

According to the medical big data file exporting method and system based on the memory mapping file and the terminal, by using the memory mapping file technology, data in the processing process is written into the memory mapping file, memory occupation is reduced, server jamming is reduced, and the throughput is improved. In the data merging process, a scheme of first sorting and then pointer traversal is adopted, the complexity of the algorithm is reduced from O (n2) to O (n), the export time consumption is greatly shortened, the matched memory occupation is reduced, the server load can be greatly reduced, and a large amount of data can be stably exported in a short time.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system and terminal for exporting medical big data files based on memory mapping files. Background Art

[0002] When hospitals conduct scientific research, they often need to import the diagnosis and treatment data of in-hospital patients into a third-party statistical analysis system for analysis. Depending on different analysis algorithms, the third-party statistical analysis system may require a large amount of patient data and splice it into one row for each patient or one row for each visit. The format may require Csv or Excel, Spss, Sas. Due to the limitation of the hospital's server hardware resources, the conventional export method may cause server freezes and memory overflows when splicing a large amount of patient data, or the file cannot be generated for a long time, which leads to export failure or long-term inability to export successfully. Summary of the invention

[0003] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a medical big data file export method, system and terminal based on memory mapping files, which is used to solve the technical problem in the prior art that due to the limitation of the hospital server hardware resources, the conventional export method may cause server freezes and memory overflows when splicing a large amount of patient data, or the file cannot be generated for a long time, thereby leading to export failure or the inability to export successfully for a long time.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a medical big data file export method based on memory mapping file, the method comprising: selecting required variables; performing group query on variable data by domain, and writing domain data of each queried variable domain into the corresponding generated memory mapping file; performing row merging on domain data of each variable domain according to target variable dimension to obtain variable domain row merging data of target variable dimension, and writing into the generated memory mapping file of target variable dimension; reading data line by line from the memory mapping file of target variable dimension to generate a final file and exporting it.

[0005] In one embodiment of the present invention, the grouping and querying of variable data by domain and writing the domain data of each queried variable domain into the corresponding generated memory mapping file includes: grouping and querying the variable data by domain and sorting according to the patient number and the consultation number to obtain the domain data of each variable domain; performing read and write operations on the domain data of each variable domain in turn to write the domain data of each variable domain into the corresponding generated memory mapping file.

[0006] In one embodiment of the present invention, the domain data of the current variable domain is read row by row and cell by cell, and the cell statistical information of each cell counted is recorded in the row information memory object of the current variable domain; wherein the cell statistical information includes: cell data volume, offset, patient number and consultation number; a memory mapping file of a corresponding file size is created according to the statistical information of each cell of the current variable domain, and the data of each cell of the current variable domain is written into the memory mapping file, and the file information of the memory mapping file is recorded into the file information memory object of the current variable domain.

[0007] In one embodiment of the present invention, the domain data of each variable domain is merged in rows according to the target variable dimension to obtain the variable domain row merged data of the target variable dimension, and the variable domain row merged data of the target variable dimension is written into the generated memory mapping file of the target variable dimension, including: performing an intra-domain data assembly operation on the domain data of each variable domain according to the target variable dimension, so as to write the obtained target variable dimension assembled data of each variable domain into the corresponding generated assembled memory mapping file; performing all domain row data assembly operations based on the assembled memory mapping files of each variable domain to obtain the variable domain row merged data of the target variable dimension, and writing the variable domain row merged data of the target variable dimension into the corresponding generated memory mapping file of the target variable dimension; wherein the types of the target variable dimension include: patient dimension and visit dimension; when the target variable dimension is a patient dimension, one target variable object is one patient; when the target variable dimension is a visit dimension, one target variable object is one visit.

[0008] In one embodiment of the present invention, the intra-domain data assembly operation includes: traversing all cell statistical information under each target variable object in the row information memory object of the current variable domain, and taking out all cell data under each target variable object in the current variable domain and assembling them into a row, obtaining the intra-domain assembled row data of each target variable object in the current variable domain, and recording all cell statistical information in each intra-domain assembled row data counted into the corresponding merged row information memory object; after the intra-domain assembled row data of all target variable objects in the current variable domain are obtained, creating an assembled memory mapping file of a corresponding file size according to all cell statistical information of each target variable object in the current variable domain, and writing the obtained intra-domain assembled row data of each target variable object into the assembled memory mapping file as the target variable dimension assembled data of the current variable domain.

[0009] In one embodiment of the present invention, the operation of assembling all domain row data includes: obtaining the target variable numbers of all variable domains, and sorting them after deduplication; traversing the merged row information memory object of each variable domain according to the sorted target variable number pointers, comparing the traversed target variable number of the cell statistical information with the current target variable number, and updating the table information when they are the same; after the traversal is completed, splicing the data of all variable domains of each target variable object into one row according to the sorted target variable numbers to obtain all domain row splicing data of each target variable object, and writing all domain row splicing data of each target variable object into the memory mapping file of the corresponding generated target variable dimension in sequence until all domain row splicing data of all target variable objects are written.

[0010] In one embodiment of the present invention, if the default export is selected, after the domain data of each variable domain queried is written into the corresponding generated memory mapping file, the data is directly read line by line from the memory mapping file of each variable domain to generate the final file and export it.

[0011] In one embodiment of the present invention, the variable domains include: a patient variable domain, a consultation variable domain, a diagnosis variable domain, a test variable domain, a surgery variable domain, and an electrocardiogram variable domain.

[0012] To achieve the above-mentioned purpose and other related purposes, the present invention provides a medical big data file export system based on memory mapping files, the system comprising: a variable selection module, used to select the required variables; a variable domain data query module, connected to the variable selection module, used to query the variable data by domain in groups, and write the domain data of each queried variable domain into the corresponding generated memory mapping file; a data merging module, connected to the variable domain data query module, used to merge the domain data of each variable domain according to the target variable dimension to obtain the variable domain row merged data of the target variable dimension, and write it into the generated memory mapping file of the target variable dimension; a file export module, connected to the data merging module, used to read the data line by line from the memory mapping file of the target variable dimension to generate the final file and export it.

[0013] To achieve the above-mentioned purpose and other related purposes, the present invention provides an electronic terminal, comprising: one or more memories and one or more processors; the one or more memories are used to store computer programs; the one or more processors are connected to the memories and are used to run the computer program to execute the medical big data file export method based on memory mapping files.

[0014] As described above, the present invention is a method, system and terminal for exporting medical big data files based on memory mapping files, which has the following beneficial effects: the present invention uses memory mapping file technology to write the data in the processing process into the memory mapping file, thereby reducing memory usage, reducing server jams and improving throughput. In the data merging process, the scheme of sorting first and then traversing with pointers is adopted, which reduces the complexity of the algorithm from O(n 2 ) is reduced to O(n), which greatly shortens the export time. Combined with the reduction in memory usage, it can greatly reduce the server load and enable large amounts of data to be exported stably in a short time. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Shown is a flowchart of a method for exporting medical big data files based on memory mapping files in one embodiment of the present invention.

[0016] Figure 2 Shown is a schematic diagram of a front-end variable selection interface in an embodiment of the present invention.

[0017] Figure 3 Shown is a flowchart of a method for exporting medical big data files based on memory mapping files in one embodiment of the present invention.

[0018] Figure 4 Shown is a structural diagram of a medical big data file export system based on memory mapping files in one embodiment of the present invention.

[0019] Figure 5 Shown is a schematic structural diagram of an electronic terminal in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] It should be noted that in the following description, reference is made to the accompanying drawings, which describe several embodiments of the present invention. It should be understood that other embodiments may also be used, and that mechanical composition, structural, electrical and operational changes may be made without departing from the spirit and scope of the present invention. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present invention is limited only by the claims of the published patents. The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0022] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a part is said to "include" a certain constituent element, unless otherwise stated, it does not exclude other constituent elements, but means that other constituent elements may be included.

[0023] The terms first, second and third mentioned herein are used to describe various parts, components, regions, layers and / or segments, but are not limited thereto. These terms are only used to distinguish a certain part, component, region, layer or segment from other parts, components, regions, layers or segments. Therefore, the first part, component, region, layer or segment described below may refer to the second part, component, region, layer or segment within the scope of the present invention.

[0024] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0025] The present invention provides a method for exporting medical big data files based on memory mapping files. By using memory mapping file technology, the data in the processing process is written into the memory mapping file, which reduces memory usage, reduces server jams, and improves throughput. In the data merging process, the scheme of sorting first and then traversing with pointers is adopted, which reduces the complexity of the algorithm from O(n 2 ) is reduced to O(n), which greatly shortens the export time. Combined with the reduction in memory usage, it can greatly reduce the server load and enable large amounts of data to be exported stably in a short time.

[0026] The following is a detailed description of the embodiments of the present invention with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.

[0027] like Figure 1 A flowchart of a method for exporting medical big data files based on memory mapping files in an embodiment of the present invention is shown.

[0028] The method comprises:

[0029] Step S1: Select the desired variables.

[0030] In one embodiment, researchers can select variables to be exported from the front end to determine the data range. The variables are grouped by domain, including: patient variable domain, visit variable domain, diagnosis variable domain, test variable domain, surgery variable domain, and electrocardiogram variable domain. Each variable domain contains multiple variables. For example, Figure 2 As shown, the variables are grouped by domains including: basic patient information, medical information, diagnosis, test, surgery, electrocardiogram, etc. The variables of basic patient information include: patient number, name, nationality, ethnicity, date of birth, telephone number, ID, etc.

[0031] Step S2: Query the variable data by domain in groups, and write the domain data of each queried variable domain into the corresponding generated memory mapping file.

[0032] In one embodiment, step S2 includes:

[0033] The variable data is grouped and queried by domain, and sorted by patient number and consultation number to obtain domain data of each variable domain; specifically, the data of each variable is queried in sequence starting from the variable domain, and the query is sorted by patient number and then consultation number to obtain domain data of each variable domain. The domain data of each variable domain includes: the data of the variable belonging to the variable domain sorted by patient number and consultation number.

[0034] The domain data of each variable domain is read and written in sequence to write the domain data of each variable domain into the corresponding generated memory mapping file.

[0035] In one embodiment, the read and write operations include:

[0036] The domain data of the current variable domain is read row by row and cell by cell, and the cell statistical information of each cell is recorded into the row information memory object of the current variable domain, and the data of each cell is written into another data object; wherein the cell statistical information includes: cell data volume (data length and size), offset (position information of cell data), patient number and consultation number;

[0037] Create a memory-mapped file of the corresponding file size according to the statistical information of each cell in the current variable domain, write the data of each cell in the current variable domain into the memory-mapped file using disk sequential writing, and record the file information of the memory-mapped file, such as file name and size, into the file information memory object of the current variable domain.

[0038] Step S3: Merge the domain data of each variable domain in rows according to the target variable dimension to obtain variable domain row merged data of the target variable dimension, and write the data into the generated memory mapping file of the target variable dimension.

[0039] In one embodiment, step S3 includes:

[0040] Performing an intra-domain data assembly operation on the domain data of each variable domain according to the target variable dimension, so as to write the obtained target variable dimension assembly data of each variable domain into the corresponding generated assembly memory mapping file respectively;

[0041] Perform all domain row data assembly operations based on the assembled memory mapping file of each variable domain, obtain variable domain row merged data of the target variable dimension, and write the variable domain row merged data of the target variable dimension into the memory mapping file of the corresponding generated target variable dimension;

[0042] Among them, the types of target variable dimensions include: patient dimension and medical consultation dimension; when the target variable dimension is the patient dimension, one target variable object is one patient; when the target variable dimension is the medical consultation dimension, one target variable object is one medical consultation.

[0043] In one embodiment, the intra-domain data assembly operation includes:

[0044] According to the row information memory object of the previous variable domain, count the total number of cells with the most records under the target variable object, and generate the number of columns of the current variable domain table based on this;

[0045] Traverse all cell statistics under each target variable object in the row information memory object of the current variable domain, calculate the record position of the data of all cells under each target variable object of the current variable domain, and take out all cell data under each target variable object of the current variable domain and assemble them into one row, obtain the assembled row data in the domain of each target variable object of the current variable domain, and record all cell statistics in each assembled row data in the statistics into the corresponding merged row information memory object;

[0046] After the in-domain assembly row data of all target variable objects in the current variable domain is obtained, an assembly memory mapping file of the corresponding file size is created according to all cell statistics of each target variable object in the current variable domain and the number of columns of the current variable domain table, and the in-domain assembly row data of each target variable object obtained is written into the assembly memory mapping file row by row as the target variable dimension assembly data of the current variable domain, and the file information of the memory mapping file is recorded into the file information memory object of the current variable domain.

[0047] In one embodiment, the whole domain row data assembly operation includes:

[0048] According to the merged row information memory object of each variable domain, the maximum number of columns in each variable domain is counted, and the number of table columns is generated based on this;

[0049] Get the target variable numbers of all variable domains, remove duplicates and sort them;

[0050] According to the pointers of the sorted target variable numbers, the merged row information memory objects of each variable domain are traversed, and the target variable numbers of the traversed cell statistical information are compared with the current target variable numbers, and the table information is updated when they are the same; specifically, according to the sorted target variable numbers, the merged row information memory objects of each variable domain are traversed, and a counter is assigned to each variable domain. Since they are all sorted by patient number, the patient row information object can be directly obtained according to the index and subscript, and compared with the currently traversed patient. If they are the same, the number is taken and the counter is increased by 1. If they are not the same, they are skipped. Compared with the nested loop traversal, the algorithm complexity can be reduced to O(n).

[0051] After the traversal is completed, the data of all variable domains of each target variable object are spliced ​​into one row according to the sorted target variable numbers to obtain the spliced ​​data of all domain rows of each target variable object, and the memory mapping file of the target variable dimension is generated according to the statistical information of the merged row information memory object of each variable domain, the number of generated table columns and the counter of each variable domain, and the spliced ​​data of all domain rows of each target variable object are written into the memory mapping file of the corresponding generated target variable dimension in sequence until the spliced ​​data of all domain rows of all target variable objects are written.

[0052] Step S4: Read data line by line from the memory-mapped file of the target variable dimension to generate the final file and export it.

[0053] In one embodiment, according to the merged row information memory object of each variable domain, data is read row by row from the memory mapping file of the target variable dimension and written into a file of a corresponding format (Excel or Csv file), and finally a final file is generated and exported.

[0054] For example, according to the merged row information memory object of each variable domain, the patient data is read row by row and written into the Excel file stream, and finally an Excel file is generated.

[0055] In one embodiment, after executing step 2, it can also be determined whether to select the default export. If the default export is selected, the data is directly read line by line from the memory mapping file of each variable domain to generate the final file and export it. If the default export is not selected, step 3 needs to be executed.

[0056] In order to better describe the medical big data file export method based on memory mapping file, it is now explained in conjunction with the following specific embodiments.

[0057] Embodiment 1: A method for exporting medical big data files based on memory mapping files. Figure 3 This is a flowchart of the medical big data file export method based on memory mapping file in this embodiment.

[0058] Researchers can select the variables to be exported from the front end to determine the data range. When exporting by default, each domain will generate an Excel worksheet (if it is CSV, it will be reflected through multiple files). When selecting a patient row, the data will be merged into a worksheet and spliced ​​into a row according to the patient dimension. Multiple data in the patient dimension will be displayed through new columns. One visit row will merge the data into a worksheet and spliced ​​into a row according to the visit dimension. Multiple data in the visit dimension will be displayed through new columns. The export process is divided into data query stage, data processing stage, and file generation stage.

[0059] The method comprises:

[0060] Step 1: Select the exported variables on the front end.

[0061] Step 2: Query data by domain group, including the following steps:

[0062] Step 21: Start querying the data by domain, sorting by patient ID first and then by visit ID;

[0063] Step 22: Read the queried data row by row and cell by cell, count the data volume of each cell, the patient's offset, and the patient number and visit number of the current row into the row information memory object, and then write the cell content into another data object.

[0064] Step 3: Write the domain data of each variable domain queried into the corresponding generated memory mapping file.

[0065] After all cells in the current domain are read, the file size is calculated based on the statistical information to create a memory-mapped file, and then the data object is written to the file using disk sequential writing, and the file information (file name, size) is recorded in the file information memory object;

[0066] In the data query stage, steps S2 and S3 query the variable domains one by one. The next stage will not be entered until all variable domains are successfully queried. Steps 2 and 3 are repeated until all variable domains are processed. When each domain query is completed, the data is written to the memory mapping file to save memory.

[0067] Step S3: Determine whether to export by default.

[0068] If the default export is selected, step S6 is executed; if the default export is not selected, step S4 is executed.

[0069] Step S4: convert one patient row / one visit row in the variable domain and write into a new memory-mapped file.

[0070] If you convert a patient row, the specific steps are as follows:

[0071] Step 411: Count the total number of records with the most patients according to the row information memory object, and generate columns and table headers accordingly;

[0072] Step 412: Traverse the patient dimension according to the row information object, calculate the location of all patient data records, take out all the cell data of the current patient, assemble them into a row and rewrite the new data object, count the data volume, offset, and current row patient number and visit number of each cell and record them into the row information memory object.

[0073] Step 413: When all patients have been processed, the file size is calculated based on the row information memory object to create a memory mapping file, and then the data object is written into the file using a sequential write method, and the file information is recorded into the file information memory object.

[0074] Repeat steps 411, 412, and 413 until all patients have been treated, and then perform subsequent steps 511 to 513.

[0075] If you want to convert a medical visit line, the specific steps are as follows:

[0076] Step 421: Count the total number of records with the most visits according to the row information memory object, and generate columns and table headers accordingly;

[0077] Step 422: Traverse the visit dimension according to the row information object, calculate the location of all data records for each visit, take out all the cell data of the current visit, assemble them into a row and rewrite them into a new data object, count the data volume, offset, and current row patient number and visit number of each cell, and record them into the row information memory object.

[0078] Step 423: When all the visits are processed, the file size is calculated based on the row information memory object to create a memory mapping file, and then the data object is written into the file using a sequential write method, and the file information is recorded into the file information memory object.

[0079] Repeat steps 421, 422, and 423 until all visits are processed, and then execute subsequent step 521.

[0080] Step 5: Merge all variable fields into a new memory-mapped file, one row for each patient / one row for each visit.

[0081] If you convert a patient row, the specific steps are as follows:

[0082] Step 511: Count the maximum number of columns in each domain according to the row information memory object, and generate columns and table headers accordingly.

[0083] Step 512: Obtain the patient numbers at the front of all domains, remove duplicates, and sort them.

[0084] Step 513: traverse the patient number, and then traverse the row information memory objects of each domain generated in step 412 according to the current patient number, set a counter for each variable domain, and use the sorted patient number to directly compare the currently traversed patient through the index. If they are the same, extract the data and add 1 to the counter, otherwise skip. This method avoids complex nested loops and optimizes the algorithm complexity to O(n). Subsequently, the cross-domain patient data is spliced ​​into a row according to the comparison results, and the row information is counted and recorded, and then written to the memory mapping file in sequence. Repeat this process until all patient data is processed, and finally count and record the detailed information of the entire memory mapping file.

[0085] If you want to convert a medical visit line, the specific steps are as follows:

[0086] Step 521: Count the maximum number of columns in each domain according to the row information memory object, and generate columns and table headers accordingly.

[0087] Step 522: Get all the front-end medical numbers of the domain, remove duplicates, and sort them.

[0088] Step 523: traverse the medical consultation number, and then traverse the row information memory objects of each domain generated in step 422 according to the current medical consultation number, set a counter for each variable domain, and use the sorted medical consultation number to directly compare the currently traversed medical consultation number through the index. If they are the same, extract the data and add 1 to the counter count, otherwise skip. This method avoids complex nested loops and optimizes the algorithm complexity to O(n). Subsequently, the cross-domain medical consultation data is spliced ​​into a row according to the comparison results, and the row information is counted and recorded, and then written into the memory mapping file in sequence. Repeat this process until all medical consultation data are processed, and finally count and record the detailed information of the entire memory mapping file.

[0089] Step 6: Read data from the memory mapped file and write it to the final file.

[0090] Similar to the principle of the above-mentioned embodiment, the present invention provides a medical big data file export system based on memory mapping files.

[0091] The following provides specific embodiments in conjunction with the accompanying drawings:

[0092] like Figure 4 A structural schematic diagram of a medical big data file export system based on memory mapping files in an embodiment of the present invention is shown.

[0093] The system comprises:

[0094] Variable selection module 1, used to select required variables;

[0095] A variable domain data query module 2, connected to the variable selection module 1, is used to query the variable data by domain in groups, and write the domain data of each queried variable domain into a corresponding generated memory mapping file;

[0096] A data merging module 3, connected to the variable domain data query module 2, is used to merge the domain data of each variable domain according to the target variable dimension to obtain the variable domain row merged data of the target variable dimension, and write it into the generated memory mapping file of the target variable dimension;

[0097] The file export module 4 is connected to the data merging module 3 and is used to read data line by line from the memory mapping file of the target variable dimension to generate a final file and export it.

[0098] It should be understood that Figure 4 The division of each module in the system embodiment is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. And these units can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some units can also be implemented in the form of processing elements calling software, and some units can be implemented in the form of hardware.

[0099] Since the implementation principle of the medical big data file export system based on memory mapping files has been described in the above embodiments, it will not be repeated here.

[0100] The medical big data file export method based on memory mapping file provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the electronic terminal, please refer to Figure 5 , is an optional hardware structure diagram of an electronic terminal 1000 provided in an embodiment of the present invention. The terminal 1000 may be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The terminal 1000 includes: at least one processor 1001, a memory 1002, at least one network interface 10010 and a user interface 1009. The various components in the device are coupled together through a bus system 1005. It can be understood that the bus system 1005 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1005 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 5 In the specification, various buses are labeled as bus systems.

[0101] The user interface 1009 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0102] It is understood that the memory 1002 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0103] The memory 1002 in the embodiment of the present invention is used to store various categories of data to support the operation of the terminal 1000. Examples of these data include: any executable program for operating on the terminal 1000, such as an operating system 10021 and an application 10022; the operating system 10021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 10022 may include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The medical big data file export method based on memory-mapped files provided in the embodiment of the present invention may be included in the application 10022.

[0104] The method disclosed in the above embodiment of the present invention can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1001 or the instruction in the form of software. The above processor 1001 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 1001 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0105] In an exemplary embodiment, the terminal 1000 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0106] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0107] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0108] Compared with the prior art, the present invention has the following specific advantages:

[0109] 1. This invention introduces memory-mapped file technology, which can directly write intermediate results into files when processing large-scale data, effectively reducing the memory usage of the program when it is running. This innovation not only alleviates the memory pressure of the server and reduces the jamming phenomenon caused by insufficient memory, but also significantly improves the data throughput of the system, ensuring efficient and stable performance.

[0110] 2. In the data merging stage, the present invention abandons the traditional nested loop traversal method and adopts an efficient strategy of sorting the data first and then using pointers to perform linear traversal. This algorithmic innovation reduces the algorithm complexity from O(n 2 ) is reduced to O(n), thereby greatly shortening the data export time and significantly reducing the server resource usage time. This advantage is particularly prominent when processing massive amounts of data, bringing users a smoother data processing experience.

[0111] In summary, the medical big data file export method, system and terminal based on memory mapping file of the present invention write the data in the processing process into the memory mapping file by using memory mapping file technology, thereby reducing memory usage, reducing server jams and improving throughput. In the data merging process, the scheme of sorting first and then traversing with pointers is adopted, which reduces the complexity of the algorithm from O(n 2 ) is reduced to O(n), which greatly shortens the export time consumption. With the reduction of memory usage, the server load can be greatly reduced, and a large amount of data can be stably exported in a short time. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has a high industrial utilization value.

[0112] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for exporting medical big data files based on memory mapping files, characterized in that: The method comprises: Select the desired variable; Query the variable data by domain in groups, and write the domain data of each variable domain queried into the corresponding generated memory mapping file; Merge the domain data of each variable domain in rows according to the target variable dimension to obtain the variable domain row merged data of the target variable dimension, and write the data into the generated memory mapping file of the target variable dimension; Read data line by line from the memory-mapped file of the target variable dimension to generate the final file and export it.

2. The method for exporting medical big data files based on memory mapping files according to claim 1, characterized in that: The querying of the variable data by domain and writing the domain data of each queried variable domain into the corresponding generated memory mapping file includes: Query the variable data by domain, sort them by patient number and visit number, and obtain the domain data of each variable domain; The domain data of each variable domain is read and written in sequence to write the domain data of each variable domain into the corresponding generated memory mapping file.

3. The method for exporting medical big data files based on memory mapping files according to claim 2, characterized in that: The read and write operations include: The domain data of the current variable domain is read row by row and cell by cell, and the cell statistical information of each cell is recorded in the row information memory object of the current variable domain; wherein the cell statistical information includes: cell data volume, offset, patient number and consultation number; A memory mapping file of a corresponding file size is created according to the statistical information of each cell of the current variable domain, and the data of each cell of the current variable domain is written into the memory mapping file, and the file information of the memory mapping file is recorded into the file information memory object of the current variable domain.

4. The method for exporting medical big data files based on memory mapping files according to claim 3 is characterized in that: The step of merging the domain data of each variable domain according to the target variable dimension to obtain variable domain row merged data of the target variable dimension, and writing the data into the generated memory mapping file of the target variable dimension includes: Performing an intra-domain data assembly operation on the domain data of each variable domain according to the target variable dimension, so as to write the obtained target variable dimension assembly data of each variable domain into the corresponding generated assembly memory mapping file respectively; Perform all domain row data assembly operations based on the assembled memory mapping file of each variable domain, obtain variable domain row merged data of the target variable dimension, and write the variable domain row merged data of the target variable dimension into the memory mapping file of the corresponding generated target variable dimension; Among them, the types of target variable dimensions include: patient dimension and medical consultation dimension; when the target variable dimension is the patient dimension, one target variable object is one patient; when the target variable dimension is the medical consultation dimension, one target variable object is one medical consultation.

5. The method for exporting medical big data files based on memory mapping files according to claim 4, characterized in that: The intra-domain data assembly operation includes: Traverse all cell statistics under each target variable object in the row information memory object of the current variable domain, and take out all cell data under each target variable object of the current variable domain and assemble them into one row, obtain the assembled row data in the domain of each target variable object of the current variable domain, and record all cell statistics in each assembled row data in the statistics into the corresponding merged row information memory object; After the in-domain assembly row data of all target variable objects in the current variable domain are obtained, an assembly memory mapping file of the corresponding file size is created based on all cell statistics of each target variable object in the current variable domain, and the in-domain assembly row data of each target variable object obtained is written into the assembly memory mapping file as the target variable dimension assembly data of the current variable domain.

6. The method for exporting medical big data files based on memory mapping files according to claim 5, characterized in that: The whole domain row data assembly operation includes: Get the target variable numbers of all variable domains, remove duplicates and sort them; According to the sorted target variable number pointers, the merged row information memory objects of each variable domain are traversed, and the target variable number of the traversed cell statistical information is compared with the current target variable number, and the table information is updated if they are the same; After the traversal is completed, all variable domain data of each target variable object are spliced ​​into one row according to the sorted target variable numbers to obtain all domain row splicing data of each target variable object, and all domain row splicing data of each target variable object are written into the memory mapping file of the corresponding generated target variable dimension in sequence until all domain row splicing data of all target variable objects are written.

7. The method for exporting medical big data files based on memory mapping files according to claim 1, characterized in that: If default export is selected, after the domain data of each variable domain queried is written into the corresponding generated memory-mapped file, the data is directly read line by line from the memory-mapped file of each variable domain to generate the final file and export it.

8. The method for exporting medical big data files based on memory mapping files according to claim 1, characterized in that: The variable domains include: a patient variable domain, a consultation variable domain, a diagnosis variable domain, a test variable domain, a surgery variable domain and an electrocardiogram variable domain.

9. A medical big data file export system based on memory mapping file, characterized in that: The system comprises: Variable selection module, used to select the required variables; A variable domain data query module, connected to the variable selection module, for querying the variable data by domain in groups, and writing the domain data of each queried variable domain into a corresponding generated memory mapping file; A data merging module, connected to the variable domain data query module, is used to merge the domain data of each variable domain according to the target variable dimension to obtain the variable domain row merged data of the target variable dimension, and write it into the generated memory mapping file of the target variable dimension; The file export module is connected to the data merging module and is used to read data line by line from the memory mapping file of the target variable dimension to generate a final file and export it.

10. An electronic terminal, characterized in that: include: one or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors, connected to the memory, are configured to run the computer program to perform the method according to any one of claims 1 to 8.

Images