Method, device and equipment for synchronous extraction of heterogeneous files based on different storage modes

Abstract

The present invention discloses a method, device and equipment for synchronous extraction of heterogeneous files based on different storage methods, the method comprising: performing data retrieval from a main system database according to a preset data index extraction condition to generate a data index to be extracted; storing the data index to be extracted in a memory database to generate data to be consumed; starting a synchronous data consumption process, creating a polling task list based on the data to be consumed, and synchronously performing file polling searches in storage media of different storage methods, wherein the different storage methods include: directory-level traditional file storage and system-level network object storage; outputting data extraction results according to a preset data extraction method to complete synchronous extraction of heterogeneous files from storage media of different storage methods. The present invention realizes unified scheduling of heterogeneous files through the above steps, improves the extraction and control efficiency of heterogeneous files, and thus improves data extraction efficiency and software and hardware resource utilization.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device and equipment for synchronously extracting heterogeneous files based on different storage modes. Background Art

[0002] Traditional storage methods refer to file management based on the directory level, such as DAS and NAS storage methods. To extract files, you need to traverse the directories one by one, and their access capabilities depend on local I / O, CPU performance, etc. Object storage methods are based on system-level file management, such as S3 protocol object storage methods. Files on object storage cannot be directly extracted, and you need to configure the account and object storage bucket information before you can download and extract them. When users need to extract relevant data for offline data review, due to the different storage methods of these two storage methods, it is impossible to achieve synchronous file extraction and unified scheduling, and data extraction efficiency is low. Summary of the invention

[0003] The present invention provides a method, device and equipment for synchronous extraction of heterogeneous files based on different storage modes, which realizes synchronous extraction and unified scheduling of heterogeneous files on storage media with different storage modes, and improves data extraction efficiency.

[0004] A method for synchronously extracting heterogeneous files based on different storage modes, comprising:

[0005] Retrieve data from the main system database according to preset data index extraction conditions and generate the data index to be extracted;

[0006] The index of the data to be extracted is stored in the memory database to generate data to be consumed;

[0007] Starting a synchronous data consumption process, creating a polling task list based on the data to be consumed, and synchronously performing file polling searches in storage media of different storage methods, wherein the different storage methods include: directory-level traditional file storage and system-level network object storage;

[0008] Output the data extraction results according to the preset data extraction method to achieve synchronous extraction of heterogeneous files from storage media with different storage methods.

[0009] According to a method for synchronous extraction of heterogeneous files based on different storage methods provided by the present invention, the data index extraction conditions include: using a data extraction tool for static extraction; correspondingly, pre-storing the query conditions in a text file and / or a list file, and importing them into the data extraction tool, and realizing data retrieval from the system database through the data extraction tool.

[0010] According to a method for synchronously extracting heterogeneous files based on different storage methods provided by the present invention, the data index extraction condition also includes: using a data extraction tool for dynamic extraction; correspondingly, based on multiple query options set on the interface of the data extraction tool, after determining the query condition based on one or more of the query options, data retrieval from the system database is implemented through the data extraction tool.

[0011] According to a method for synchronously extracting heterogeneous files based on different storage modes provided by the present invention, the query options include: business classification, insurance type and category, time interval and file category.

[0012] According to a method for synchronously extracting heterogeneous files based on different storage modes provided by the present invention, data is retrieved from a main system database according to a preset data index extraction condition to generate a data index to be extracted, including:

[0013] Create a data cache task;

[0014] Cache the data index extraction condition to the data cache task;

[0015] Based on a preset timing scheduling strategy, the data index extraction condition satisfying the timing scheduling strategy is retrieved from the data cache task;

[0016] Performing data retrieval from the main system database based on the retrieved data index extraction condition to generate the data index to be extracted;

[0017] The data index extraction condition for completing data retrieval is marked as completed extraction in the data cache task.

[0018] According to a method for synchronously extracting heterogeneous files based on different storage methods provided by the present invention, the data extraction methods include: full extraction, relative path extraction and category extraction;

[0019] If you choose to extract all the files, configure the output root directory and output the extracted files to the output root directory;

[0020] If you choose to extract by relative path, the files to be extracted will be generated into relative paths and output to the list file;

[0021] If category extraction is selected, an output root directory is configured, and after the extracted files are output to the output root directory, only files of the specified file type are extracted according to the specified file type.

[0022] According to a method for synchronous extraction of heterogeneous files based on different storage methods provided by the present invention, it also includes: obtaining a first data volume of data to be consumed in the memory database and a real-time remaining data volume of data to be consumed in the polling task list, and calculating the ratio of the real-time remaining data volume to the first data volume for real-time monitoring of data extraction progress.

[0023] The present invention also provides a device for synchronously extracting heterogeneous files based on different storage modes, comprising:

[0024] The main system database retrieval module is used to retrieve data from the main system database according to the preset data index extraction conditions and generate the data index to be extracted;

[0025] A memory database storage module, used to store the index of the data to be extracted into the memory database to generate data to be consumed;

[0026] A synchronous data consumption module, used to start a synchronous data consumption process, create a polling task list based on the data to be consumed, and synchronously perform file polling and searching in storage media of different storage methods, wherein the different storage methods include: directory-level traditional file storage and system-level network object storage;

[0027] The data extraction module is used to output the data extraction results according to the preset data extraction method to achieve the synchronous extraction of heterogeneous files from storage media with different storage methods.

[0028] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for synchronously extracting heterogeneous files based on different storage methods as described above is implemented.

[0029] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for synchronously extracting heterogeneous files based on different storage methods as described in any one of the above is implemented.

[0030] The present invention provides a method, device and equipment for synchronous extraction of heterogeneous files based on different storage methods. The method retrieves data from a main system database according to preset data index extraction conditions to generate a data index to be extracted; stores the data index to be extracted in a memory database to generate data to be consumed; starts a synchronous data consumption process, creates a polling task list based on the data to be consumed, and synchronously performs file polling and search in storage media of multiple different storage methods such as directory-level traditional file storage and system-level network object storage; outputs data extraction results according to a preset data extraction method to complete the synchronous extraction of heterogeneous files from storage media of different storage methods, thereby realizing unified scheduling of heterogeneous files, improving the extraction and management efficiency of heterogeneous files, and thus improving data extraction efficiency and software and hardware resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A schematic diagram of a flow chart of a method for synchronously extracting heterogeneous files based on different storage modes provided by an embodiment of the present invention;

[0033] Figure 2 An example diagram of a data extraction tool provided by an embodiment of the present invention;

[0034] Figure 3 An execution flow chart for an actual application scenario provided by an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of the structure of a device for synchronously extracting heterogeneous files based on different storage modes provided by an embodiment of the present invention;

[0036] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0037] Reference numerals:

[0038] 21: Main system database retrieval module; 22: Memory database storage module; 23: Synchronous data consumption module; 24: Data extraction module. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be described clearly and completely in combination with the specific embodiments and drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in the present invention should be the usual meanings understood by people with ordinary skills in the technical field to which the present invention belongs. The words "one", "a", "a", "the" and the like involved in the present invention do not indicate a quantitative limitation and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in the present invention are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in the present invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0041] The following are explanations of relevant terms in the present invention:

[0042] DAS (Direct Access Storage) refers to connecting a storage device directly to a computer via a SCSI interface or a fiber channel. The components of the DAS storage structure mainly include three parts: host / server, network, and storage device; its working method is: each host / server manages its own data storage location and backup method. Specifically, it relies on the host's operating system to implement data I / O reading and writing, data management, and data backup;

[0043] NAS (Network Attached Storage) refers to connecting storage devices to a group of computers through a standard network topology (such as Ethernet). NAS is a component-level storage method that focuses on helping workgroups and department-level organizations address the need for rapidly increasing storage capacity. An engineering team that needs to share large CAD documents is a typical example. The components of the NAS storage structure mainly include four parts: host / server, network, storage server, and storage device. Its working method is: the host / server is connected to the storage server through the network to facilitate data writing and reading; the storage server is responsible for the specific data writing and reading process, and also controls the data backup and recovery operations.

[0044] S3 protocol object storage (Amazon Simple Storage Service) is a public cloud storage service with remote access, massive users and unlimited expansion. Web application developers can use it to store digital assets, including pictures, videos, music and documents. S3 provides a RESTful API to programmatically interact with the service. It can be used to store and retrieve any amount of data anywhere on the Web at any time. The data storage structure of S3 is divided into two layers, the first layer is Bucket, and the second layer is Object. Object is stored in Bucket. S3 can create multiple Buckets, and each Bucket can store multiple Objects. So Bucket is like a folder, and Object is more like a file in a file.

[0045] H2 database is an embedded (memory level) database developed in Java. It is just a class library, that is, there is only one jar file, which can be directly embedded in the project. H2 database is also called in-memory database because it supports creating databases and tables in memory. So if we use the memory mode of H2 database, the databases and tables we create are only saved in memory. Once the server is restarted, the databases and tables in memory will no longer exist.

[0046] Embodiment 1

[0047] Reference Figure 1 As shown, this embodiment provides a method for synchronously extracting heterogeneous files based on different storage modes, including:

[0048] Step S1: searching data from the main system database according to preset data index extraction conditions to generate a data index to be extracted;

[0049] Among them, the data index extraction conditions include: using data extraction tools for static extraction and dynamic extraction;

[0050] In static extraction, the query conditions are stored in a text file and / or a list file in advance and imported into the data extraction tool, and data retrieval is performed from the system database through the data extraction tool. Specifically, the query conditions such as the order number and the ID card number are stored in a text file in txt format or a list file in excel format in advance, and are imported through the import button set on the data extraction tool interface. The data extraction tool retrieves data from the system database in a static extraction manner according to the query conditions.

[0051] During dynamic extraction, correspondingly, based on multiple query options set on the interface of the data extraction tool, after determining the query conditions based on one or more query options, the data extraction tool is used to retrieve data from the system database. Specifically, Figure 2 As shown, the interface of the data extraction tool is provided with query options such as business classification, insurance type and category, time period, and file category, including required options and non-required options. One or more query options can be selected according to the actual user. After the data extraction tool determines the query conditions based on the selected query options, it retrieves data from the system database in a dynamic extraction manner according to the query conditions.

[0052] Furthermore, since the system database stores a massive amount of data, a database paging search strategy is used for data search in the system database to avoid the search service being blocked due to excessively large amounts of data being queried each time.

[0053] Step S2: Store the index of the data to be extracted into the memory database to generate the data to be consumed;

[0054] Specifically, the in-memory database is the H2 database, which is an embedded database built into the application. After the data extraction tool is deployed and started, the H2 database is also started. The table is created through the mapping file; the jdbc interface is called through the mapping bean object to connect to the H2 database, and insert (insert) and consume (select) after the connection is successful. The index of the data to be extracted stored in the H2 database includes: business type, order number, version, storage location, task scheduling id, task status and other data. Before storing the index of the data to be extracted in the in-memory database, the in-memory database is automatically initialized to clear its stored data.

[0055] Step S3: Start the synchronous data consumption process, create a polling task list based on the data to be consumed, and synchronously perform file polling and searching in storage media of different storage methods, wherein the different storage methods include: directory-level traditional file storage and system-level network object storage;

[0056] Specifically, when extracting heterogeneous files, the data to be consumed recorded in the H2 database is searched for. A polling task list is created based on the data to be consumed and mounted on storage media with different storage methods. The synchronous data consumption process uses parallel processing to poll and search files from different storage media. For location that is directory-level traditional file storage (DAS, NAS), the directory traversal is used to search and extract files, such as the path "home / web / app / 2023 / 3 / 12 / batch / "; for location that is system-level network object storage (S3 protocol object storage), the key of the spliced ​​file is used for search and extraction, and the splicing is in the form of "domain name + bucket + prefix + file name" to complete the query of heterogeneous files on storage media with different storage methods.

[0057] Step S4: outputting the data extraction result according to the preset data extraction method to achieve synchronous extraction of heterogeneous files from storage media with different storage methods.

[0058] Among them, data extraction methods include: full extraction, relative path extraction and category extraction;

[0059] If you choose to extract all the files, configure the output root directory and output the extracted files to the output root directory;

[0060] If you choose to extract by relative path, the files to be extracted will be generated into relative paths and output to the list file;

[0061] If you select category extraction, configure the output root directory, output the extracted files to the output root directory, and then extract only the files of the corresponding file type according to the specified file type.

[0062] Specifically, the data extraction tool provides three data extraction methods: full extraction, relative path extraction and category extraction. The background of the data extraction tool provides configuration functions. The full extraction method can output the extracted files to the specified directory according to the configured output root directory, while the category extraction method specifies the file type of the extracted files on the basis of the full extraction, such as only extracting files of the corresponding file type, such as PDF and image formats. Relative path extraction supports file relative path extraction, that is, a relative path is generated for the file to be extracted and output to an Excel file, realizing file extraction in accordance with the requirements specified by the regulatory agency, facilitating regulatory agency review.

[0063] In this embodiment, step S1 specifically includes:

[0064] According to the preset data index extraction conditions, data is retrieved from the main system database to generate the data index to be extracted, including:

[0065] Create a data cache task;

[0066] Cache the data index extraction conditions to the data cache task;

[0067] Based on the preset timing scheduling strategy, the data index extraction condition satisfying the timing scheduling strategy is retrieved from the data cache task;

[0068] Retrieve data from the main system database based on the retrieved data index extraction condition to generate an index of data to be extracted;

[0069] The data index extraction condition that has completed data retrieval is marked as completed extraction in the data cache task.

[0070] Specifically, the task status of the data index extraction condition is set, including: wait (waiting: waiting for scheduled scheduling), run (executing: extracting scheduled scheduling), done (ended: scheduled scheduling completed), error (scheduled scheduling has an exception), pause (scheduled scheduling has been paused), rewait (scheduled scheduling is waiting for execution again). Scheduled retrieval performs a scan every 5 seconds. When the task status is pause, it remains paused and waits for the next schedule; when the task status is wait or rewait, extraction begins; when the task status is set to run, the extraction adopts paging extraction, extracting 100 each time until the extraction is completed; if an exception occurs during the extraction process, the task status is set to error; after the extraction is completed, the task status is set to done.

[0071] When all data index extraction conditions are queried from the system database, the synchronous data consumption process is automatically triggered.

[0072] In this embodiment, it also includes: obtaining a first data volume of data to be consumed in the memory database and a real-time remaining data volume of data to be consumed in the polling task list, and calculating the ratio of the real-time remaining data volume to the first data volume for real-time monitoring of data extraction progress.

[0073] Specifically, based on the polling task list mounted on the storage medium of different storage methods, the real-time remaining data volume of the data to be consumed in the polling task list is obtained, and the ratio of the real-time remaining data volume to the first data volume of the data to be consumed stored in the memory database before the synchronous data consumption process is started is calculated, and the progress of data extraction is displayed in the form of a progress bar on the interface of the data extraction tool to facilitate real-time monitoring by the user. The interface of the data extraction tool also provides a pause function to pause or start the data extraction progress according to the actual situation; and a refresh function for refreshing the data extraction progress in real time.

[0074] In summary, Figure 3The figure shows an execution flow chart of an embodiment of the present invention in an actual application scenario, such as the annual routine data spot check and verification by the regulatory agency (CIRC), the upload summary of image data of each business system at the end of the year, and other application scenarios. The user uses a data extraction tool to perform static extraction or dynamic extraction to retrieve data from the main system database and generate an index of data to be extracted; the index of data to be extracted is stored in the memory database to generate data to be consumed; the data extraction tool starts a synchronous data consumption process, creates a polling task list based on the data to be consumed and mounts it on storage media with different storage methods, uses parallel processing to poll and search files from different storage media, and outputs data extraction results according to a preset data extraction method to complete the synchronous extraction of heterogeneous files from storage media with different storage methods, thereby realizing unified scheduling of heterogeneous files, improving the extraction and management efficiency of heterogeneous files, and thus improving data extraction efficiency and software and hardware resource utilization.

[0075] Embodiment 2

[0076] Based on the same inventive idea as the above method, Figure 4 As shown, this embodiment provides a device for synchronously extracting heterogeneous files based on different storage modes, including:

[0077] The main system database search module 21 is used to search for data from the main system database according to the preset data index extraction conditions and generate the data index to be extracted;

[0078] The memory database storage module 22 is used to store the index of the data to be extracted into the memory database to generate the data to be consumed;

[0079] The synchronous data consumption module 23 is used to start the synchronous data consumption process, create a polling task list based on the data to be consumed, and synchronously perform file polling and searching in storage media of different storage methods, wherein the different storage methods include: directory-level traditional file storage and system-level network object storage;

[0080] The data extraction module 24 is used to output the data extraction result according to a preset data extraction method, so as to realize the synchronous extraction of heterogeneous files from storage media of different storage methods.

[0081] Specifically, the main system database retrieval module 21 includes: a static extraction setting unit, used to store the preset query conditions in a text file and / or a list file; an import unit, used to import the text file and / or the list file into the data extraction tool; a dynamic extraction setting unit, used to determine the query conditions based on one or more query options set on the interface of the data extraction tool; a retrieval unit, used to retrieve data from the main system database according to the preset data index extraction conditions; a to-be-extracted data index generation unit, used to generate the to-be-extracted data index;

[0082] The memory database storage module 22 includes: an initialization unit for automatically initializing the memory database to clear the data stored therein; a storage unit for storing the index of the data to be extracted into the memory database to generate the data to be consumed;

[0083] The synchronous data consumption module 23 includes: a polling task list creation unit, which is used to create a polling task list based on the data to be consumed; a mounting unit, which is used to mount the polling task list on storage media with different storage methods; a synchronous data consumption process start unit, which is used to start the synchronous data consumption process and perform file polling and search from different storage media in a parallel processing manner; a first file search unit, which is used to use traversal directory search and extraction of files for storage media with traditional file storage methods at the directory level; and a second file search unit, which is used to use the key of the spliced ​​file to search and extract for storage media with system-level network object storage methods.

[0084] The data extraction module 24 includes: a first mode extraction unit, which is used to output the extracted files to the output root directory according to the configured output root directory; a second mode extraction unit, which is used to generate a relative path for the files to be extracted and output it to a list file; a third mode extraction unit, which is used to output the extracted files to the output root directory according to the configured output root directory, and then, according to the specified file type, only extract files of the corresponding file type.

[0085] The device also includes: a synchronous data consumption process automatic triggering module, which is used to automatically trigger and start the synchronous data consumption process when all data index extraction conditions are queried from the system database;

[0086] And, a data extraction progress monitoring module is used to obtain the real-time remaining data volume of the data to be consumed in the polling task list, calculate the ratio of the real-time remaining data volume to the second quantity, so as to monitor the data extraction progress in real time.

[0087] The implementation process of the functions and effects of each module in the above-mentioned device is specifically detailed in the implementation process of the corresponding steps in the above-mentioned method, so the relevant parts can be referred to the partial description of the method embodiment, which will not be repeated here.

[0088] The device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. There may be other division methods in actual implementation. The functional modules in the embodiments may all be integrated into one processor, or each module may be a separate device, or two or more modules may be integrated into one device. The functional modules in the embodiments may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0089] Embodiment 3

[0090] Reference Figure 5 As shown, this embodiment provides an electronic device, which includes: a processor (processor) 310, a communication interface (Communications Interface) 320, a memory (memory) 330 and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call the logic instructions in the memory 330, and the processor 310 executes the heterogeneous file synchronization extraction method based on different storage methods described in the above method embodiment, and the method includes:

[0091] Retrieve data from the main system database according to preset data index extraction conditions and generate the data index to be extracted;

[0092] Store the index of the data to be extracted into the memory database to generate the data to be consumed;

[0093] Start the synchronous data consumption process, create a polling task list based on the data to be consumed, and synchronously perform file polling and searching in storage media of different storage methods, where the different storage methods include: directory-level traditional file storage and system-level network object storage;

[0094] Output the data extraction results according to the preset data extraction method to achieve synchronous extraction of heterogeneous files from storage media with different storage methods.

[0095] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0096] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the heterogeneous file synchronization extraction method based on different storage methods described in the above method embodiment, the method comprising:

[0097] Retrieve data from the main system database according to preset data index extraction conditions and generate the data index to be extracted;

[0098] Store the index of the data to be extracted into the memory database to generate the data to be consumed;

[0099] Start the synchronous data consumption process, create a polling task list based on the data to be consumed, and synchronously perform file polling and searching in storage media of different storage methods, where the different storage methods include: directory-level traditional file storage and system-level network object storage;

[0100] Output the data extraction results according to the preset data extraction method to achieve synchronous extraction of heterogeneous files from storage media with different storage methods.

[0101] Embodiment 4

[0102] This embodiment provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for synchronously extracting heterogeneous files based on different storage methods described in the above method embodiment is implemented. The method includes:

[0103] Retrieve data from the main system database according to preset data index extraction conditions and generate the data index to be extracted;

[0104] Store the index of the data to be extracted into the memory database to generate the data to be consumed;

[0105] Start the synchronous data consumption process, create a polling task list based on the data to be consumed, and synchronously perform file polling and searching in storage media of different storage methods, where the different storage methods include: directory-level traditional file storage and system-level network object storage;

[0106] Output the data extraction results according to the preset data extraction method to achieve synchronous extraction of heterogeneous files from storage media with different storage methods.

[0107] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0108] The devices and media provided in the embodiments of the present invention correspond one-to-one to the methods, and therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0109] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or method. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or method including the elements.

[0110] The above are only embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed in the present invention.

Claims

1. A method for synchronously extracting heterogeneous files based on different storage modes, characterized in that: include: Retrieve data from the main system database according to preset data index extraction conditions and generate the data index to be extracted; The index of the data to be extracted is stored in the memory database to generate data to be consumed; Starting a synchronous data consumption process, creating a polling task list based on the data to be consumed, and synchronously performing file polling searches in storage media of different storage methods, wherein the different storage methods include: directory-level traditional file storage and system-level network object storage; Output data extraction results according to the preset data extraction method to achieve synchronous extraction of heterogeneous files from storage media with different storage methods; The step of retrieving data from the main system database according to the preset data index extraction condition to generate the data index to be extracted includes: Create a data cache task; Cache the data index extraction condition to the data cache task; Based on a preset timing scheduling strategy, the data index extraction condition satisfying the timing scheduling strategy is retrieved from the data cache task; Performing data retrieval from the main system database based on the retrieved data index extraction condition to generate the data index to be extracted; The data index extraction condition for completing data retrieval is marked as completed extraction in the data cache task.

2. The method for synchronously extracting heterogeneous files based on different storage modes according to claim 1 is characterized in that: The data index extraction conditions include: using a data extraction tool for static extraction; correspondingly, pre-storing the query conditions in a text file and / or a list file, and importing them into the data extraction tool, and realizing data retrieval from the main system database through the data extraction tool.

3. The method for synchronously extracting heterogeneous files based on different storage modes according to claim 1 is characterized in that: The data index extraction conditions also include: using a data extraction tool for dynamic extraction; correspondingly, based on multiple query options set on the interface of the data extraction tool, after determining the query conditions based on one or more of the query options, data retrieval from the main system database is achieved through the data extraction tool.

4. The method for synchronously extracting heterogeneous files based on different storage modes according to claim 3 is characterized in that: The query options include: business classification, insurance type and category, time period and file category.

5. The method for synchronously extracting heterogeneous files based on different storage modes according to claim 1, characterized in that: The data extraction methods include: full extraction, relative path extraction and category extraction; If you choose to extract all the files, configure the output root directory and output the extracted files to the output root directory; If you choose to extract by relative path, the files to be extracted will be generated into relative paths and output to the list file; If category extraction is selected, an output root directory is configured, and after the extracted files are output to the output root directory, only files of the specified file type are extracted according to the specified file type.

6. The method for synchronously extracting heterogeneous files based on different storage modes according to claim 1, characterized in that: Also includes: The first data volume of the data to be consumed in the memory database and the real-time remaining data volume of the data to be consumed in the polling task list are obtained, and the ratio of the real-time remaining data volume to the first data volume is calculated for real-time monitoring of data extraction progress.

7. A device for synchronously extracting heterogeneous files based on different storage modes, characterized in that: include: The main system database retrieval module is used to retrieve data from the main system database according to the preset data index extraction conditions and generate the data index to be extracted; A memory database storage module, used to store the index of the data to be extracted into the memory database to generate data to be consumed; A synchronous data consumption module, used to start a synchronous data consumption process, create a polling task list based on the data to be consumed, and synchronously perform file polling and searching in storage media of different storage methods, wherein the different storage methods include: directory-level traditional file storage and system-level network object storage; The data extraction module is used to output the data extraction result according to the preset data extraction method, so as to realize the synchronous extraction of heterogeneous files from storage media of different storage methods; Wherein, the main system database retrieval module is specifically used for: The step of retrieving data from the main system database according to the preset data index extraction condition to generate the data index to be extracted includes: Create a data cache task; Cache the data index extraction condition to the data cache task; Based on a preset timing scheduling strategy, the data index extraction condition satisfying the timing scheduling strategy is retrieved from the data cache task; Performing data retrieval from the main system database based on the retrieved data index extraction condition to generate the data index to be extracted; The data index extraction condition for completing data retrieval is marked as completed extraction in the data cache task.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for synchronously extracting heterogeneous files based on different storage methods as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for synchronously extracting heterogeneous files based on different storage methods as described in any one of claims 1 to 6 is implemented.

Images