A mass file archiving and intelligent scheduling method, device and medium
By implementing a four-stage management strategy for medical image files, including time-based management, lossless compression, and partitioned storage, the problem of high image file storage consumption was solved, and storage space was optimized while maintaining data access efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 中电通商数字技术(上海)有限公司
- Filing Date
- 2023-07-14
- Publication Date
- 2026-05-12
AI Technical Summary
In the medical imaging industry, unstructured image files occupy a large amount of storage space, and compression affects data access efficiency. Existing technologies struggle to reduce storage consumption while maintaining data access timeliness.
A four-stage management strategy is adopted to manage unstructured image files by time period, including T0 stage storing and saving structured data as individual images, T1 stage lossless compression and archiving, T2 stage splitting storage and clearing memory indexes, and T3 stage generating temporary index data. Data access is ensured to be unaffected through the main index and read/write configuration files.
It effectively reduces storage space consumption by 50%-60% while maintaining data access performance and ensuring the timeliness of data query and download.
Smart Images
Figure CN117056542B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of document management technology, and in particular to a method, device, and medium for massive document archiving and intelligent scheduling. Background Technology
[0002] In the medical imaging industry, unstructured image files occupy storage units measured in terabytes (TB), especially digitized pathology images, which can reach staggering petabyte (PB) levels annually in leading hospitals. The structured data that matches this unstructured data is also stored in the tens or hundreds of millions. Furthermore, data has a time-sensitivity factor; the likelihood of data being reused after a certain period decreases significantly, and it may even cease to be used altogether under special circumstances. Technically, general solutions address query and retrieval speeds through non-relational databases and object storage, while cold and hot storage technologies and the selection of low-cost storage media save on hardware costs.
[0003] The most effective way to reduce storage consumption is compression. Actual calculations show that compressing medical image files can reduce storage consumption by about 50%-60%. However, compression will prevent business systems from using them in the usual way. Summary of the Invention
[0004] To address the aforementioned problems, this invention aims to solve the issues described above. One objective of this invention is to provide a method for massive file archiving and intelligent scheduling that solves the problems described above. This method re-archives unstructured image file data, reducing storage consumption while also reducing the amount of paired structured data, without affecting the timeliness of data access and download.
[0005] The solution adopted in this embodiment is: a method for massive file archiving and intelligent scheduling, including the following steps:
[0006] Step S10: Divide data with the same attributes into four time periods: T0, T1, T2, and T3 for management;
[0007] Step S20: Store the unstructured data of stage T0 as a single image, save the structured data paired with each image, store the structured data in the memory index library, and create the master index;
[0008] Step S30: For the data entering stage T1, perform lossless compression and archiving on the image set corresponding to the main index, and convert the structure dataset corresponding to the main index into a readable and writable configuration file;
[0009] Step S40: For the data entering stage T2, the data set corresponding to the main index is stored in a separate database, and the memory index database is cleared, as well as the image set corresponding to the main index.
[0010] Step S50: For data that exceeds stage T2 but is still needed by the business, regenerate temporary unstructured data for a single image and memory index database data paired with a single image, and delete the temporary memory index database data and unstructured data after entering stage T3.
[0011] The usable data files have the following characteristics. In step S10, data with the same attributes include the following characteristics: they belong to the same image set for inspection, and the image set is used simultaneously when it is used.
[0012] The time periods are defined as follows: In step S10, the time when the first unstructured image file of a check is entered into the business system is defined as T0; the timeliness time of medical image data is defined as T1, after which the retrieval frequency and access frequency of image data reach a low point; the original data processing time after re-archiving is defined as T2; and the time after querying and using the data after stage T2 is defined as T3.
[0013] A preferred embodiment of the present invention is that, in step S20, only one master index is generated for structured data with the same attributes, and the time of the master index is updated every time a piece of structured data is added to the database.
[0014] A preferred technical solution is that, during stages T0, T1, and T2, the application system directly accesses the memory index library to obtain the download path and downloads the data through the download path.
[0015] The specific data processing in step S30 includes: Step S31, for unarchived data whose access time to the main index exceeds T1, all unstructured data and matching structured data belonging to the check represented by the main index are extracted; Step S32, the extracted unstructured data is uniformly compressed into a compressed file for storage, and the path is written into the main index; Step S33, the extracted structured data is converted into a custom readable and writable configuration file with relative paths as the core for storage, and the path is written into the main index.
[0016] Furthermore, after steps S32 and S33 are completed, the readable and writable configuration file and compressed file are downloaded through the main index, the compressed file is decompressed, and the corresponding file is generated according to the readable and writable configuration file and verified with the original data to ensure that the data is not distorted; the readable and writable configuration file is a JSON configuration file or an XML configuration file.
[0017] To minimize storage space usage, in step S40, after archiving the data in stage T1, the processing of the original data includes: deleting the original image set corresponding to the main index; backing up the structured data corresponding to the main index in a separate database for future verification if there are special needs for the data, and deleting the original structured data after the backup is completed; and deleting the structured data related to the memory index database.
[0018] For data entering stage T3, in step S50, when the application system queries the checklist, it checks whether the corresponding data exists in the synchronous memory index library. If it does not exist, it directly generates a main index sensing event. After receiving the sensing event, it will obtain the compressed file and readable / writable configuration file based on the main index data, and generate temporary memory index library data and unstructured data according to the readable / writable configuration file for use by the corresponding application.
[0019] This invention provides a massive file archiving and intelligent scheduling device for the operation of the aforementioned massive file archiving and intelligent scheduling method.
[0020] This invention provides a medium for massive file archiving and intelligent scheduling, including a computer-readable storage medium storing computer-executable instructions, which are used to implement the method when executed by a processor.
[0021] The massive file archiving and intelligent scheduling method of this invention has the following technical effects:
[0022] 1. The massive file archiving and intelligent scheduling method of this application includes the following steps: Step S10: Data with the same attributes are divided into four time periods: T0, T1, T2, and T3 for management; different management strategies are adopted for files in different stages to ensure that performance is not affected to the greatest extent, while greatly compressing storage space.
[0023] 2. Step S20: Store the unstructured data of stage T0 as individual images, save the structured data paired with each image, store the structured data in the memory index library, and establish the master index; establish the correspondence to facilitate searching and ensure that the data use in this stage is not affected by any images.
[0024] 3. Step S30: For the data entering stage T1, the image set corresponding to the main index is compressed and archived without loss, and the structure dataset corresponding to the main index is converted into a readable and writable configuration file. Compression can greatly reduce storage space, while conversion into a readable and writable configuration file can maintain the characteristic of easy reading.
[0025] 4. Step S40: For data entering stage T2, the data set corresponding to the main index is stored in a separate database, and the memory index database is cleared, as well as the image set corresponding to the main index. For data entering stage T2, database sharding and clearing can minimize memory usage. At the same time, since the file usage frequency is greatly reduced after entering stage T2, it will not have a significant impact on data usage.
[0026] 5. Step S50: For data that exceeds stage T2 but is still needed by the business, regenerate temporary unstructured data for a single image and memory index database data paired with that image. Then, delete the temporary memory index database data and unstructured data after entering stage T3. By regenerating temporary files and then clearing them after use, subsequent use will not be affected, and there will be no memory consumption.
[0027] Other features and advantages of the invention will become clear when reading the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description
[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. In these drawings, similar reference numerals are used to denote similar elements. The drawings described below are some embodiments of the invention, but not all embodiments. Other drawings will be readily available to those skilled in the art based on these drawings without any inventive effort.
[0029] Figure 1 This is a schematic diagram illustrating the overall principle of the massive file archiving and intelligent scheduling method provided in a specific embodiment of the present invention; Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.
[0031] Example 1
[0032] The following description, in conjunction with the accompanying drawings and embodiments, details the method for massive file archiving and intelligent scheduling.
[0033] like Figure 1The diagram shown illustrates the principle of file scheduling in this embodiment. This embodiment uses data from the medical imaging industry as an example. The unstructured image data associated with a single examination has two characteristics: first, these image sets naturally share the same attribute, belonging to the same examination; second, these image sets are used simultaneously when they are used. The entire technical implementation path for a single examination, addressing these two characteristics, is divided into four stages.
[0034] This invention provides a method for massive file archiving and intelligent scheduling, comprising the following steps: Step S10: Data with the same attributes are managed in four time periods: T0, T1, T2, and T3. Each time period is defined as follows: In Step S10, T0 is defined as the time when the first unstructured image file of a check is entered into the business system; T1 is defined as the timeliness time of medical image data, after which the retrieval and access frequencies of image data reach a low point; T2 is defined as the processing time of the original data after re-archiving; and T3 is defined as the time after a period of querying and using the data after T2. T1 and T2 are compared based on the time of the main index generated in T0. T3 is a time comparison based on the time of generating temporary data. A preferred technical solution is that, during T0, T1, and T2, the application system directly accesses the memory index to obtain the download path and downloads the data through the download path.
[0035] In the medical imaging industry, examination results are typically required to be stored for 15 to 30 years. However, due to quality control and mutual recognition requirements, data used more frequently has a limited lifespan, such as within 30 days (T2). Data from other years is only archived and not accessed unless there are special circumstances. By optimizing the process using the five steps described above, only detailed data and in-memory indexes from the past 30 days need to be stored. Data from other years is stored as compressed data, significantly improving system query performance and reducing storage space consumption by 50%-60%.
[0036] Step S20: For unstructured data in stage T0, store it as a single image, save the structured data paired with each image, store the structured data in the memory index library, and create a master index. A preferred embodiment of this invention is that in step S20, only one master index is generated for structured data with the same attributes, and the time of the master index is updated each time a piece of structured data is added to the library. This data can be used simultaneously in subsequent applications. By creating only one master index, space can be saved to the maximum extent, and there is no impact on data retrieval. It also provides the convenience of one-time compression for subsequent compression, ensuring convenient retrieval after compression and a high degree of compression.
[0037] Step S30: For data entering stage T1, the image set corresponding to the main index is losslessly compressed and archived, and the structured dataset corresponding to the main index is converted into a readable and writable configuration file; the readable and writable configuration file can be a JSON configuration file or an XML configuration file, and this embodiment preferably uses a JSON configuration file. The specific processing of data in step S30 includes: Step S31, for unarchived data whose access time to the main index exceeds T1, all unstructured data and matching structured data belonging to the check represented by the main index are extracted; Step S32, the extracted unstructured data is uniformly compressed into a compressed file for storage, and the path is written into the main index; Step S33, the extracted structured data is converted into a custom JSON configuration file with relative paths as the core for storage, and the path is written into the main index. After time T1, the retrieval frequency and access frequency of image data reach a low point. Under this compression method, the data entering stage T1 is compressed to reduce storage space.
[0038] Furthermore, after steps S32 and S33 are completed, the JSON configuration file and compressed file are downloaded via the main index. The compressed file is then decompressed, and the corresponding file is generated based on the JSON configuration file. This data is then verified against the original data to ensure that the data is not distorted. This ensures that the compressed file will not affect usability.
[0039] Step S40: For data entering stage T2, the structured dataset corresponding to the main index is sharded and stored in separate databases. Simultaneously, the in-memory index is cleared, and the image set corresponding to the main index is also cleared. To minimize storage space usage, in step S40, after data archiving in stage T1, the processing of the original data includes: deleting the original image set corresponding to the main index; backing up the structured data corresponding to the main index in separate databases for future verification if there are special data needs, and deleting the original structured data after the backup is complete; and deleting the related structured data in the in-memory index. Deleting the original image set reduces a significant amount of space usage, while backing up the structured data ensures that data is not lost or damaged.
[0040] Step S50: For data that exceeds stage T2 but is still needed by the business, temporary unstructured data for a single image and memory index database data paired with that image are regenerated. Upon entering stage T3, the temporary memory index database data and unstructured data are deleted. For data entering stage T3, in step S50, when the application system queries the checklist, it synchronizes the memory index database data to see if corresponding data exists. If not, a main index sensing event is generated directly. Upon receiving the sensing event, the compressed file and JSON configuration file are obtained based on the main index data, and temporary memory index database data and unstructured data are generated according to the JSON configuration file for use by the corresponding application. When the data is needed subsequently, a temporary file is generated based on the previously compressed file, allowing for direct access. The file is deleted upon entering stage T3 after use, preventing unnecessary space usage.
[0041] Example 2
[0042] This invention provides a massive file archiving and intelligent scheduling device for running the massive file archiving and intelligent scheduling method described in the claims.
[0043] The data acquisition module is used to acquire image data from the medical data system.
[0044] The time-series module manages the acquired image data into four time periods: T0, T1, T2, and T3. The T1 and T2 periods are compared with the time of the main index generated in T0. T3 is a time comparison based on the generated temporary data.
[0045] Storage module: For unstructured data in stage T0, it stores the data as individual images and saves the structured data paired with each image. The structured data is then stored in the memory index library, and a main index is created. For data entering stage T2, the structured dataset corresponding to the main index is stored in a separate database. At the same time, the memory index library is cleared, and the image set corresponding to the main index is also cleared.
[0046] Compression module: For data entering stage T1, lossless compression and archiving are performed on the image set corresponding to the main index, and the structured dataset corresponding to the main index is converted into a readable and writable configuration file.
[0047] Decompression scheduling module: For data that exceeds stage T2 but is still needed by the business, regenerate temporary unstructured data of a single image and memory index library data paired with a single image, and delete the temporary memory index library data and unstructured data after entering stage T3.
[0048] Example 3
[0049] This invention provides a medium for massive file archiving and intelligent scheduling, including a computer-readable storage medium storing computer-executable instructions, which are used to implement the method when executed by a processor.
[0050] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the article or apparatus that includes that element.
[0051] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. The present invention has been described in detail with reference to preferred embodiments. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications and substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for massive file archiving and intelligent scheduling, characterized in that: Includes the following steps: Step S10: Divide data with the same attributes into four time periods: T0, T1, T2, and T3 for management; Step S20: Store the unstructured data of stage T0 as a single image, save the structured data paired with each image, store the structured data in the memory index library, and create the master index; Step S30: For the data entering stage T1, perform lossless compression and archiving on the image set corresponding to the main index, and convert the structure dataset corresponding to the main index into a readable and writable configuration file; Step S40: For the data entering stage T2, the data set corresponding to the main index is stored in a separate database, and the memory index database is cleared, as well as the image set corresponding to the main index. Step S50: For data that exceeds stage T2 but is still needed by the business, regenerate temporary unstructured data for a single image and memory index database data paired with a single image, and delete the temporary memory index database data and unstructured data after entering stage T3. In step S10, data with the same attributes include the following characteristics: they belong to the same image set for inspection, and the image set is used simultaneously when it is used. In step S10, the time when the first unstructured image file of a single inspection is entered into the business system is defined as T0. Define the time-sensitivity period T1 for medical image data. After time T1, the retrieval frequency and access frequency of image data reach a low point. Define the original data processing time after re-archiving as T2; define the time after querying and using the data after stage T2 as T3.
2. The method for massive file archiving and intelligent scheduling as described in claim 1, characterized in that: In step S20, only one primary index is generated for structured data with the same attributes, and the time of the primary index is updated every time a piece of structured data is added to the database.
3. The method for massive file archiving and intelligent scheduling as described in claim 1 or 2, characterized in that: When in stages T0, T1, and T2, the application system directly accesses the memory index to obtain the download path and downloads the data through the download path.
4. The method for massive file archiving and intelligent scheduling as described in claim 1, characterized in that: The specific processing of data in step S30 includes: Step S31, for unarchived data whose access time to the main index exceeds T1, all unstructured data and matching structured data belonging to the check represented by the main index are retrieved; Step S32: Compress the extracted unstructured data into a single compressed file and store it, then write the path into the main index; Step S33: The extracted structured data is converted into a custom readable and writable configuration file storage based on relative paths, and the paths are written into the main index.
5. The method for massive file archiving and intelligent scheduling as described in claim 4, characterized in that: After steps S32 and S33 are completed, the readable and writable configuration file and compressed file are downloaded through the main index. The compressed file is decompressed and the corresponding file is generated according to the readable and writable configuration file and verified with the original data to ensure that the data is not distorted. The readable and writable configuration file can be a JSON configuration file or an XML configuration file.
6. The method for massive file archiving and intelligent scheduling as described in claim 1, characterized in that: In step S40, after the data is archived in stage T1, the processing of the original data includes: deleting the original image set corresponding to the main index; backing up the structured data corresponding to the main index in a separate database for future verification if there are special needs for the data, and deleting the original structured data after the backup is completed; and deleting the structured data related to the memory index database.
7. The method for massive file archiving and intelligent scheduling as described in claim 1, characterized in that: In step S50, when the application system queries the checklist, it checks whether the corresponding data exists in the synchronous memory index database. If it does not exist, it directly generates a main index sensing event. Upon receiving a sensing event, the system will retrieve the compressed file and readable / writable configuration file based on the main index data, and generate temporary memory index library data and unstructured data according to the readable / writable configuration file for use by the corresponding application.
8. A massive file archiving and intelligent scheduling device, characterized in that: Used for running the massive file archiving and intelligent scheduling method according to any one of claims 1-7.
9. A medium for massive file archiving and intelligent scheduling, characterized in that: The invention includes a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.