Data exporting method and device for information system, storage medium and computer program

Abstract

The invention relates to a data exporting method and device of an information system, a storage medium and a computer program, and the method comprises the steps: receiving an exporting request, initiated by a user, of query data, responding to the exporting request, analyzing the exporting request, and obtaining the exporting data size of to-be-exported data, and the to-be-exported data is the data required to be queried by the user. Whether the exported data volume is larger than a preset data volume threshold value or not is detected, and the preset data volume threshold value is a preset numerical value and used for determining whether multi-thread query is conducted or not. When the exported data volume is greater than the preset data volume threshold value, it is indicated that the exported data volume is relatively large, the duration required for processing the to-be-exported data by a single thread is relatively large, the to-be-exported data is split into a preset number of groups of first sub-data, the data volume of each piece of first sub-data is smaller than the exported data volume, and the duration required for processing each piece of first sub-data is reduced.

Description

Technical Field

[0001] The present disclosure relates to the technical field of information management, and in particular to a data export method, device, storage medium and computer program of an information system. Background Art

[0002] The asset management information system is a management system for the entire life cycle of Sinopec's enterprise assets. It contains a large amount of asset-related data, and its business functions such as scrapping and inventory include large data import and export scenarios.

[0003] Existing technologies use a synchronous method to directly retrieve data from the database based on query conditions and then export it. This can cause a series of problems in scenarios with large amounts of data, such as millions of items. For example, when a user initiates a request, if the amount of data requested is large, the memory resources are too large, the response to the request is often slow, and it is easy to cause the request to time out or be interrupted. Summary of the invention

[0004] The present disclosure provides a data export method, device, storage medium and computer program for an information system to improve the response speed and reduce the probability of request timeout or interruption.

[0005] In a first aspect, the present disclosure provides a data export method of an information system, comprising:

[0006] receiving an export request;

[0007] Responding to the export request, parsing the export request, and obtaining the export data volume of the data to be exported;

[0008] Detecting whether the amount of exported data is greater than a preset data amount threshold;

[0009] When the amount of exported data is greater than the preset data amount threshold, the data to be exported is divided into a preset number of groups of first sub-data, a preset number of threads are used to query the preset number of groups of first sub-data in the source database respectively, and the first sub-data obtained by the query are written into a preset number of memory pages respectively to obtain a preset number of groups of first page data, wherein each thread queries a group of the first sub-data, and each thread writes the first page data obtained by the query into a memory page;

[0010] The first page data are merged to obtain the exported data.

[0011] In some embodiments, the step of writing the first divided data obtained by query into a preset number of memory pages respectively includes:

[0012] Writing the first data obtained from the query into a preset number of memory pages respectively;

[0013] Detecting the amount of data of the first data segment written to the memory segment to obtain the amount of successful data;

[0014] Storing the successful data volume in a memory database;

[0015] The method further comprises:

[0016] storing the exported data volume in the memory database;

[0017] In response to receiving a progress query request, reading the successful data volume and the exported data volume from the memory database;

[0018] The ratio of the successful data volume to the exported data volume is calculated to obtain the export progress.

[0019] In some embodiments, the method further comprises:

[0020] When the amount of exported data is less than or equal to the preset data amount threshold, detecting whether the data to be exported is empty data;

[0021] When the data to be exported is empty data, outputting the data to be exported;

[0022] When the data to be exported is not empty data, the data to be exported is written into the memory page to obtain the exported data.

[0023] In some embodiments, the step of using a preset number of threads to query a preset number of groups of first sub-data in the source database respectively, and writing the queryed data into a preset number of memory pages respectively, to obtain a preset number of groups of first page data includes:

[0024] Based on the stream processing component, a preset number of threads are used to query the first data of a preset number of groups in the source database respectively, and the queried data are written into a preset number of memory pages respectively to obtain the first page data of the preset number of groups.

[0025] In some embodiments, the method further comprises:

[0026] receiving an import request;

[0027] Responding to the import request, parsing the import request, and obtaining the import data volume of the data to be imported;

[0028] Detecting whether the amount of imported data is greater than a preset data amount threshold;

[0029] When the amount of imported data is greater than the preset data amount threshold, the data to be imported is divided into a preset number of groups of second sub-data, a preset number of threads are used to query the preset number of groups of second sub-data in the source document respectively, and the second sub-data obtained by the query are written into a preset number of memory pages respectively to obtain a preset number of groups of second page data, wherein each thread queries a group of the second sub-data, and each thread writes the second page data obtained by the query into a memory page;

[0030] The second page data are merged to obtain imported data, and the imported data is stored in the source database.

[0031] In some embodiments, the method comprises:

[0032] Obtaining a source document, and determining a corresponding form configuration according to the storage requirements of the business data of the source document; wherein the form configuration includes configuration parameters and a corresponding configuration format;

[0033] Traversing the source document, determining the area where each data corresponding to each of the configuration parameters is located in the source document as the verification area;

[0034] Check whether the data in the verification area conforms to the corresponding configuration format;

[0035] When the data in the verification area conforms to the corresponding configuration format, the corresponding data in the source document is stored in the source database;

[0036] When the data in the verification area does not conform to the corresponding configuration format, the corresponding data in the source document and the exception information generated by the verification are stored in the object storage.

[0037] In some embodiments, the configuration format includes that the data format is not empty and / or the data length is within a preset range.

[0038] In a second aspect, the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above aspect.

[0039] In a third aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the above aspects when executed by a processor.

[0040] In a fourth aspect, the present disclosure provides a computer program product, including a computer program / instructions, which implements the steps of the method described in the above aspects when the computer program is executed by a processor.

[0041] The present disclosure provides a data export method, device, storage medium and computer program for an information system, which receives an export request for query data initiated by a user, responds to the export request, parses the export request, and obtains the export data volume of the data to be exported, where the data to be exported is the data that the user needs to query. It is detected whether the export data volume is greater than a preset data volume threshold, which is a preset value used to determine whether to perform a multi-threaded query. When the export data volume is greater than the preset data volume threshold, it means that the export data volume is large, and the time required for a single thread to process the data to be exported is long. The data to be exported is divided into the first sub-data of a preset number of groups, and the data volume of each first sub-data is less than the export data volume, and the time required to process each first sub-data is reduced. A preset number of threads are used to query a preset number of groups of first divided data in the source database respectively, each thread queries a group of first divided data, and the first page data queried by each thread is written into a memory page. Since the amount of first divided data queried and written by each thread is less than the amount of exported data, the time required for each thread to process the first divided data is reduced, and the processing of the first divided data by each thread occurs simultaneously. The query and write time of each first divided data is the maximum time for processing the first divided data in each thread. When the preset number is larger, the amount of first divided data required to be processed by each thread is smaller, the maximum time for processing the first divided data in each thread is also smaller, the response speed is faster, and the probability of request timeout or interruption is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present disclosure will be described in more detail below based on embodiments and with reference to the accompanying drawings:

[0043] Figure 1 A flowchart of a data export method of an information system provided in an embodiment of the present disclosure;

[0044] Figure 2 A schematic diagram of a data export scenario of an information system provided by an embodiment of the present disclosure;

[0045] Figure 3 Another schematic diagram of a data export scenario of an information system provided by an embodiment of the present disclosure;

[0046] Figure 4 A data flow diagram of a method for exporting and importing data of an information system provided by an embodiment of the present disclosure;

[0047] Figure 5 Another data flow diagram of a method for exporting and importing data of an information system provided by an embodiment of the present disclosure;

[0048] Figure 6 A schematic diagram of a data import scenario of an information system provided in an embodiment of the present disclosure.

[0049] In the drawings, the same reference numerals are used for the same components and the drawings are not drawn to scale. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, and to fully understand and implement how the present disclosure applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. The embodiments of the present disclosure and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0052] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] Embodiment 1

[0054] Figure 1 The following is a flow chart of a method for exporting data from an information system provided by an embodiment of the present disclosure. Figures 1 to 3 As shown, a data export method of an information system includes:

[0055] Step 110: receiving an export request.

[0056] In this embodiment, the export request is generated by detecting the query operation initiated by the user on the source database, detecting the query requirement information input by the user, and the query requirement information is used to determine the query requirement of the user. Different query requirements generate different query requirement information. The export request is used to request to export the data matching the query requirement information from the source database to an export document, which is used to write the export data and store the export document in the object storage. The user can access the object storage to download the export document, thereby browsing the export data.

[0057] Step 120: respond to the export request, parse the export request, and obtain the export data volume of the data to be exported.

[0058] In this embodiment, a query operation on the source database is triggered in response to an export request, the export request is parsed, and the query requirement information of the export request is determined. Different export requests correspond to different query requirement information. A query is performed according to the query requirement information of the export request, the data in the source database is traversed, and data that meets the query requirement information is screened out as data to be exported. The amount of exported data is the amount of data that meets the query requirement information. The more data that meets the query requirement information, the larger the corresponding amount of exported data.

[0059] Step 130: Detect whether the amount of exported data is greater than a preset data amount threshold.

[0060] In this embodiment, the preset data volume threshold is a preset value, which is used as a comparison object for the exported data volume, and whether multi-thread processing is required is determined according to the comparison result of the two. When the exported data volume is greater than the preset data volume threshold, it means that the data to be exported is large, and the time required for single-thread processing of the exported data is long; when the exported data volume is less than or equal to the preset data volume threshold, it means that the data to be exported is small, and the time required for single-thread processing of the exported data is short.

[0061] In one embodiment, the method further comprises:

[0062] When the amount of exported data is less than or equal to the preset data amount threshold, detecting whether the data to be exported is empty data;

[0063] When the data to be exported is empty data, outputting the data to be exported;

[0064] When the data to be exported is not empty data, the data to be exported is written into the memory page to obtain the exported data. In this embodiment,.

[0065] Step 140, when the amount of exported data is greater than the preset data amount threshold, split the data to be exported into a preset number of groups of first sub-data, use a preset number of threads to query the preset number of groups of first sub-data in the source database respectively, write the first sub-data obtained by the query into a preset number of memory pages respectively, and obtain a preset number of groups of first page data, wherein each thread queries a group of the first sub-data, and each thread writes the first page data obtained by the query into a memory page.

[0066] In this embodiment, when the amount of exported data is greater than the preset data amount threshold, the data to be exported is relatively large, and multi-threaded processing is started. The data to be exported is split into a preset number of groups of first sub-data, each first sub-data is part of the data to be exported, and different first sub-data correspond to different parts of the data to be exported, and the data after the first sub-data are merged is the data to be exported. The preset number is used to determine the number of groups after the data to be exported is divided, that is, the number of first sub-data. The larger the preset number, the more first sub-data are obtained by splitting, the smaller the data volume of each first sub-data is, and the faster the processing speed is. The preset number is also used to determine the number of threads to be opened. The number of threads is equal to the number of first sub-data, and each thread processes a unique first sub-data respectively, and different threads process different first sub-data. Each thread queries the corresponding first sub-data in the source database and writes the first sub-data obtained by the query into a memory page. The number of memory pages is equal to the number of first divided data, each first divided data is written into a unique memory page, different first divided data correspond to different memory pages, when a first divided data is written into a memory page, a first page data is obtained, the first page data corresponds to the first divided data, and the number of the first page data and the number of the first divided data are both preset numbers.

[0067] In one embodiment, the step of writing the first divided data obtained by query into a preset number of memory pages respectively includes:

[0068] Writing the first data obtained from the query into a preset number of memory pages respectively;

[0069] Detecting the amount of data of the first data segment written to the memory segment to obtain the amount of successful data;

[0070] Storing the successful data volume in a memory database;

[0071] The method further comprises:

[0072] storing the exported data volume in the memory database;

[0073] In response to receiving a progress query request, reading the successful data volume and the exported data volume from the memory database;

[0074] The ratio of the successful data volume to the exported data volume is calculated to obtain the export progress.

[0075] In this embodiment, when each thread writes the first sub-data obtained by the query into a preset number of memory pages respectively, the data volume of the first sub-data queried and written into the memory page gradually increases until the corresponding first sub-data is all written into the corresponding memory page, and the thread completes the reading and writing tasks of the first sub-data. In order to provide the progress of data export, when each thread writes the first sub-data obtained by the query into a preset number of memory pages respectively, the data volume of the first sub-data written into the memory page in each thread is detected to obtain the successful data volume of each thread. The successful data volume is the data volume of the first sub-data that the thread completes the query and writes. When the successful data volume reaches the data volume of the corresponding first sub-data, it means that the thread completes the processing task of the corresponding first sub-data. The successful data volume of each thread is stored in the memory database. When the export progress needs to be queried, the memory database can be accessed to read the successful data volume of each thread. In order to more conveniently display the export progress, after the export data volume is stored in the memory database, when the progress needs to be displayed to the user, a progress query request can be automatically generated or the user initiates a progress query request. The progress query request is used to request to query the export progress of the data to be exported. In response to the progress query request, the memory database is accessed, the successful data volume of each thread is read, and the ratio of the sum of the successful data volume of each thread to the exported data volume is calculated to obtain the export progress, which is the overall export progress. In order to show the export situation in more detail, the ratio of the successful data volume of each thread to the exported data volume of the part of the data to be exported obtained by each corresponding thread from the split data to be exported is calculated to obtain the export progress of each thread, so that the export progress of each thread can be displayed to the user. Furthermore, the memory database can be a redis database.

[0076] In one embodiment, the step of using a preset number of threads to query a preset number of groups of first sub-data in the source database respectively, and writing the query-obtained data into a preset number of memory pages respectively, to obtain a preset number of groups of first page data includes:

[0077] Based on the stream processing component, a preset number of threads are used to query the first data of a preset number of groups in the source database respectively, and the data obtained by the query are written into a preset number of memory pages respectively to obtain the first page data of the preset number of groups. In this embodiment, the stream processing component can be an Alibaba EasyExcel stream processing component, and the AlibabaEasyExcel stream processing component is used to support data stream operations, avoid excessive memory usage during query and writing, and reduce the probability of resource recovery.

[0078] Step 150: merge the first page data to obtain exported data.

[0079] In this embodiment, when each thread completes the query and write task of the corresponding first data, the complete first page data is obtained, and the first page data processed by each thread is the data to be exported required by the export request. In order to facilitate user download and query, the first page data are merged to obtain a copy of the exported data.

[0080] like Figures 4 to 6 As shown, in one embodiment, the method further includes:

[0081] receiving an import request;

[0082] Responding to the import request, parsing the import request, and obtaining the import data volume of the data to be imported;

[0083] Detecting whether the amount of imported data is greater than a preset data amount threshold;

[0084] When the amount of imported data is greater than the preset data amount threshold, the data to be imported is divided into a preset number of groups of second sub-data, a preset number of threads are used to query the preset number of groups of second sub-data in the source document respectively, and the second sub-data obtained by the query are written into a preset number of memory pages respectively to obtain a preset number of groups of second page data, wherein each thread queries a group of the second sub-data, and each thread writes the second page data obtained by the query into a memory page;

[0085] The second page data are merged to obtain imported data, and the imported data is stored in the source database.

[0086] In this embodiment, the import request is generated by detecting that the user initiates an import data operation to the source database, and the data to be imported input by the user is detected. The import operation of the data to be imported is triggered in response to the import request. The amount of imported data is the amount of data to be imported. The preset data amount threshold is a preset value, which is used as a comparison object for the amount of imported data. Whether multi-thread processing is required is determined based on the comparison results of the two. When the amount of imported data is greater than the preset data amount threshold, it means that there are more data to be imported, and the time required for single-thread processing of the data to be imported is longer. When the amount of imported data is less than or equal to the preset data amount threshold, it means that there are fewer data to be imported, and the time required for single-thread processing of the data to be imported is shorter. When the amount of imported data is greater than the preset data amount threshold, there are more data to be imported, and multi-thread processing is started. The data to be imported is divided into second sub-data of a preset number of groups, each second sub-data is part of the data to be imported, and different second sub-data correspond to different parts of the data to be imported. The data after the second sub-data are merged is the data to be imported. The preset number is used to determine the number of groups after the data to be imported is divided, that is, the number of second sub-data. The larger the preset number, the more second sub-data is obtained by splitting, the smaller the data volume of each second sub-data is, and the faster the processing speed is. The preset number is also used to determine the number of threads to be opened. The number of threads is equal to the number of second sub-data. Each thread processes a unique second sub-data respectively, and different threads process different second sub-data. Each thread queries the corresponding second sub-data in the source database and writes the second sub-data obtained by the query into a memory page. The number of memory pages is equal to the number of second sub-data. Each second sub-data is written into a unique memory page. Different second sub-data correspond to different memory pages. When a second sub-data is written into a memory page, a second page data is obtained. The second page data corresponds to the second sub-data. The number of second page data and the number of second sub-data are both the preset number. When each thread completes the query and write task of the corresponding second sub-data, a complete second page data is obtained. The second page data processed by each thread is the data to be imported required by the import request. In order to facilitate user download and query, each second page data is merged to obtain a copy of the imported data.

[0087] like Figures 4 to 6 As shown, in one embodiment, the method includes:

[0088] Obtaining a source document, and determining a corresponding form configuration according to the storage requirements of the business data of the source document; wherein the form configuration includes configuration parameters and a corresponding configuration format;

[0089] Traversing the source document, determining the area where each data corresponding to each of the configuration parameters is located in the source document as the verification area;

[0090] Check whether the data in the verification area conforms to the corresponding configuration format;

[0091] When the data in the verification area conforms to the corresponding configuration format, the corresponding data in the source document is stored in the source database;

[0092] When the data in the verification area does not conform to the corresponding configuration format, the corresponding data in the source document and the exception information generated by the verification are stored in the object storage.

[0093] In this embodiment, the source document stores data to be imported. Different data to be imported are different business data. Different business data have different storage requirements, and the corresponding configuration formats that need to be met are different. The configuration format is the format requirement that needs to be met when storing business data. For example, the configuration format is a pure digital format, and the string length must be a preset character length. The configuration formats corresponding to two business data in the same source document may be the same or different. The form configuration is used to determine the configuration format of each business data in the source document. Each source document corresponds to a form configuration. The correspondence between the source document and the form configuration is predetermined. The type of business data stored in the source document is predetermined. Correspondingly, according to the verification requirements, the form configuration can be set in advance for the source document, and the correspondence between the identifier of the source document and the identifier of the form configuration is established. After obtaining the source document, the identifier of the source document is detected, and the form configuration is determined according to the correspondence. Since the data to be imported may contain multiple business data, it may be necessary to verify only some of the business data. The configuration parameter is the parameter name of the business data in the source document that needs to be verified. Each configuration parameter corresponds to a configuration format, which is determined according to the storage requirements of the configuration parameter. The configuration format is the format requirement that must be met when the business data corresponding to the configuration parameter is imported. Different configuration parameters may have different format requirements, so different configuration formats may correspond to different configuration parameters.

[0094] In this embodiment, the source document is traversed, and the area where each business data corresponding to the configuration parameter is located is queried as the verification area. The verification area is the area in the source document where the business data that needs to be format-verified should be stored. Whether the business data in the verification area conforms to the corresponding configuration format is detected. Each business data corresponds to a parameter name, i.e., a configuration parameter. The configuration parameter corresponding to the business data can be used to determine the configuration format that the business data needs to meet when it is imported, and whether each business data conforms to the corresponding configuration format is detected. When the business data conforms to the corresponding configuration format, the corresponding business data verification is successful and meets the storage requirements. When all business data in the verification area conform to the corresponding configuration format, all business data stored in the source document are imported into the source database. When the business data does not conform to the corresponding configuration format, the corresponding business data verification fails and does not meet the storage requirements, and abnormal information is generated. The abnormal information is the reason for the verification failure and / or the storage location of the business data that failed the verification in the source document. When there is a business data in the verification area that does not conform to the corresponding configuration format, the corresponding data in the source document and the abnormal information generated by the verification are stored in the object storage. The object storage can be, for example, OSS. When the user cannot import data, the object storage OSS can be accessed to query the reason for the import failure.

[0095] In one embodiment, the configuration format includes that the data format is not empty and / or the data length is within a preset range. In this embodiment, in order to more flexibly verify various business data, the configuration format can be a situation where the data format is not empty and / or the data length is within a preset range. When the configuration format is that the data format is not empty, it is detected whether the corresponding verification area of ​​the configuration parameter has no data stored. When the corresponding verification area has no data stored, the data format is empty, the configuration format is not satisfied, the verification fails, and an abnormal information is generated. When the configuration format is that the data length is within the preset range, the data length of the business data stored in the corresponding verification area of ​​the configuration parameter is detected, and it is determined whether the data length is within the preset range. When the data length is not within the preset range, the configuration format is not satisfied, the verification fails, and an abnormal information of the business data is generated.

[0096] like Figures 4 to 6 As shown, further, the step of writing the second divided data obtained by the query into a preset number of memory pages respectively includes:

[0097] Writing the second divided data obtained by query into a preset number of memory pages respectively;

[0098] Detecting the amount of the second data segment written into the memory segment to obtain the amount of successfully imported data;

[0099] Storing the successfully imported data volume in a memory database;

[0100] The method further comprises:

[0101] Storing the imported data volume in the memory database;

[0102] In response to receiving a progress query request, reading the successfully imported data volume and the imported data volume from the memory database;

[0103] The ratio of the amount of successfully imported data to the amount of imported data is calculated to obtain the import progress.

[0104] Furthermore, the method further comprises:

[0105] When the amount of imported data is less than or equal to the preset data amount threshold, detecting whether the data to be imported is empty data;

[0106] When the data to be imported is empty data, output a prompt that the stored data is empty data;

[0107] When the data to be imported is not empty data, the data to be imported is written into the memory page to obtain the imported data.

[0108] Furthermore, the step of using a preset number of threads to query a preset number of groups of second divided data in the source database respectively, and writing the queryed data into a preset number of memory pages respectively, to obtain a preset number of groups of second page data includes:

[0109] Based on the stream processing component, a preset number of threads are used to query a preset number of groups of second divided data in the source database respectively, and the queried data are written into a preset number of memory pages respectively to obtain a preset number of groups of second page data.

[0110] In this embodiment, the data export method of the information system receives an export request for query data initiated by a user, responds to the export request, parses the export request, and obtains the export data volume of the data to be exported, where the data to be exported is the data that the user needs to query. It is detected whether the export data volume is greater than a preset data volume threshold, and the preset data volume threshold is a pre-set value used to determine whether to perform a multi-threaded query. When the export data volume is greater than the preset data volume threshold, it means that the export data volume is large, and the time required for a single thread to process the export data is long. The export data is divided into the first sub-data of a preset number of groups, and the data volume of each first sub-data is less than the export data volume, and the time required to process each first sub-data is reduced. A preset number of threads are used to query a preset number of groups of first divided data in the source database respectively, each thread queries a group of first divided data, and the first page data queried by each thread is written into a memory page. Since the amount of first divided data queried and written by each thread is less than the amount of exported data, the time required for each thread to process the first divided data is reduced, and the processing of the first divided data by each thread occurs simultaneously. The query and write time of each first divided data is the maximum time for processing the first divided data in each thread. When the preset number is larger, the amount of first divided data required to be processed by each thread is smaller, the maximum time for processing the first divided data in each thread is also smaller, the response speed is faster, and the probability of request timeout or interruption is reduced.

[0111] Embodiment 2

[0112] Based on the above embodiment, a data export method of an information system is provided, comprising:

[0113] Step 1: The user enters the query parameters to send a request to query the total data volume and determine whether it exceeds the threshold (whether to enable multi-threading), such as 50,000. If yes, continue. If not, determine whether it is empty. If yes, export empty data. If not, directly perform the export operation without multi-threading and multi-sheet pages. The threshold is used to control whether to enable multi-threading processing (to save system resources, for example, if you want to export 20 data, you can export directly).

[0114] Step 2: After calling the asynchronous export method, the result is returned to the user, and the user can view the current progress by opening the import / export progress page. The export operation is to export the database data to Excel through the application for the user to download. This scenario is an export scenario. At present, the query request is the query operation performed by the user entering the relevant query conditions in the front-end system, and the database queries the data through the relevant statements generated by the operation.

[0115] Step 3: Initialize the thread pool configuration, set ten threads, split the 1 million data items, and each thread is responsible for parallel processing of 100,000 data items.

[0116] Step 4: Each thread implements the task of paging query. Each page queries 100,000, for example, thread 1 queries 100,000 to 100,000, thread 2 queries 100,000 to 200,000, and so on.

[0117] Step 5: After each thread queries the corresponding data, it writes it to the corresponding sheet page. For example, thread 1 writes to sheet-1 page, thread 2 writes to sheet-2 page, and so on.

[0118] Step 6: Each thread queries data, writes to the sheet page, and writes the total number of successful operations of the current data to the redis cache. The front end can obtain the data in Redis by polling the relevant interface for display on the real-time progress page. The sheet page is in the memory of the system application. Redis is used for progress monitoring. The data is in the corresponding sheet page. The interface always exists. Only in this step can the redis data be queried and the progress bar can be updated. Polling query is to update the progress bar by calling the interface continuously.

[0119] Step 7: When all ten threads have finished writing the sheet page, merge the data streams and upload them to a document server such as OSS for users to download. OSS is only an object storage service for storing attachments. The management system is the application system that uses the project.

[0120] Step 8: Front-end progress display When the amount of data successfully obtained from the interface is the same as the total amount, the progress status becomes 100% and the download attachment button becomes operable. The progress display is displayed in the application system. The download button is also in the application system, and the download operation will access the OSS for downloading.

[0121] In this embodiment, asynchronous requests are used to solve the problems of import and export request timeout, interruption, and inability to monitor import and export progress in the case of large amounts of data. The multi-threaded + data stream method is used to read / write data to solve the problem of excessive memory resource usage resulting in unresponsive interfaces. The interface for real-time query of import and export data volume is called in a round-robin manner to solve the pain point of not being able to intuitively monitor the progress of import and export. The polling call uses the task ID to query the total number of processing and the total number of completions in Redis. The ten threads record their own number of processing and completions respectively. The polling interface is used to sum up the numbers of each thread, and then the progress is calculated uniformly. Through the automatically generated verification area, no additional development work is required, and the verification of the required fields and length of the import fields is automatically adapted.

[0122] In this embodiment, not only multithreading and asynchronous technologies are used, but also stream processing, Redis cache, and OSS storage technologies. Although multithreading plus asynchronous processing can solve timeouts, interruptions, and unresponsiveness, the performance is still far from enough in the case of over one million. It is also necessary to read and write different Sheet pages through stream processing technology to further improve execution efficiency. In addition, there are problems with asynchronous monitoring and export file tracing, so this project uses Redis cache technology to monitor the progress of import and export by using the import and export task number as the key and updating and reading the total number and number of operations in real time; through OSS storage technology, the exported Excel document is uploaded to OSS storage and associated with the import and export task number to achieve traceability download of import and export files.

[0123] In this embodiment, in order to solve the three problems of excessive resource occupation, excessive memory usage, and slow response time when exporting in scenarios with large amounts of data in the millions, three technical solutions are proposed: asynchronous + multi-threaded + data stream reading / writing. The asynchronous method can solve the problem of interface request timeout; multi-threaded uses thread pool to avoid excessive resource occupation such as frequent switching of thread context; excel reading / writing operations use Alibaba EasyExcel components to support data stream operations to avoid excessive memory occupation.

[0124] In this embodiment, compared with the previous 15 minutes to import 100,000 data, it now takes only about 3 minutes, and the performance is improved by more than 5 times. Compared with the previous service unresponsiveness when exporting 1 million data, it can now be completed within 30 minutes. The import / export function will no longer have operation timeouts and service unresponsiveness, and the operation progress supports visual viewing.

[0125] In this embodiment, an asynchronous mechanism is used for import / export, and the operation progress is displayed in real time. Import / export data processing uses thread pool technology to split the task, and each thread is responsible for paging data processing and writing the corresponding sheet page, and then merging the processing ideas and implementation. Based on the innovative use of EasyExcel component data flow, the whole process is stream-operated to avoid excessive memory usage, and finally the file stream is uploaded to the document server for download.

[0126] See also Figure 6 In one embodiment, the imported data is checked. Through the automatically generated check area, the check of the required fields and length of the imported fields is automatically adapted without additional development work. The following steps are included:

[0127] Step 1: For the import document uploaded by the user, retrieve the basic field verification information through the business type and form import and export configuration table.

[0128] Step 2: Combine the document in step 1 with the configuration table to generate an import file with a verification area.

[0129] The document is imported in the form, and then the mandatory and length restrictions of the imported fields in the form are configured. The two are combined to generate an import file with a verification area and containing the original data. For example, in the user management function, the user form configures the mandatory user name and the ID card number length of 18 digits. When the user import function is performed, a verification area for the mandatory name field and the ID card length verification of 18 will be added to the original import file. The form configuration table records the mandatory conditions and length restrictions of each field in the form. The document is imported in the form, and the document and the configuration table are combined through the corresponding relationship of the form. The integration is carried out through the form configuration table, which records the relevant information of the field. Generalization is the flexibility and configurability of the form. When the relevant information changes, you only need to update the field content of the form configuration table, and the exported information will change automatically. When you import it again, you can use the new verification rules for verification without any additional adjustments.

[0130] Step 3: Initialize 10 threads through the thread pool, process different Sheet page data respectively, and write them into memory through EasyExcel stream processing. The stream processing feature of EasyExcel can write data into the application memory in batches to avoid errors such as memory overflow.

[0131] Step 4: Each thread calls the general validation service and uses the validation rules of the validation area to perform data import validation on the data.

[0132] Step 5: For the abnormal data verified in step 4, generate an import error data document together with the abnormal information.

[0133] Step 6: Upload the error document in step 5 to the OSS storage service for tracing and downloading.

[0134] Step 7: For the data that has been successfully verified in step 4, write the data into the database DB. For the data provided by the user, import the correct data into the DB, and only the data that failed to be imported will be provided to the user through OSS for download and then modified according to the error information and imported again.

[0135] Step 8: Store the amount of data successfully written and abnormal in step 7 into the cache Redis. Redis is independent of the database. The execution subject is the application, and the number of successes and exceptions during the execution process is written to Redis.

[0136] Step 9: Monitor the import progress by writing the data written to Redis in step 8 and the total number to be imported. Redis will not be associated with the database.

[0137] In this embodiment, in the import process, the data comes from the file, and through the processing of the application, the correct data is written into the database DB, the abnormal data is written into the document, and the total number, success number, and abnormal number in Redis are updated. The document of abnormal data will be uploaded to OSS for users to download. In the export process, the data comes from the database DB, and through the processing of the application, the exported data is written into the document, and the total number, success number, and abnormal number in Redis are updated. The document of exported data will be uploaded to OSS for users to download.

[0138] In this embodiment, Redis monitors the progress. OSS stores and keeps traces of export tasks. Originally, the code was developed separately to verify the data. Because the traditional import function does not have a verification area, it can only be verified separately by obtaining the column level of the imported file. For example, if the length of the third column cannot exceed 50, the code needs to obtain the data of the third column, and then some code logic is needed to determine whether it is greater than 50 to complete the import verification. Through the verification rules in this verification area, the verification logic is automatically executed through the verification service. For general data import functions, the required verification and length verification of the fields require separate development of code for verification of different templates and different fields, which is relatively cumbersome. This embodiment uses the automatically generated verification area, without the need for additional development work, and automatically adapts for verification.

[0139] In this embodiment, import is similar to export, that is, it is the reverse. The data of multiple sheet pages of the imported document are read separately by threads and then inserted into the database. The query parameter will execute the count() method to obtain the total amount of data queried. This data is the total number of corresponding query parameters of the source table data. The asynchronous method is to return the export task status (successful creation, creation exception) and the task number of the successfully created export task. Follow up the task number to monitor the export progress in Redis. The data of the export task will generate an Excel file and upload it to OSS. The user obtains the file address of OSS through the task number for downloading. The Redis data is updated by the task number, where the "total number" is a constant value m, assuming 100000. In addition, the "current processing number" is a variable n, which is updated in real time during the export process. The front end is scheduled by polling and displays the progress percentage by calculating n / m. The number of core threads is 10, the maximum number of threads is 10, and the number of queues is 10. The number of threads is fixed at 10. The number of items processed by each thread is 750,000 / 10, which is equal to 75,000. 1 million is also an example number, which is actually the number of source table data items queried by the query parameters. Each thread follows the query parameters and the number of pages to query. The purpose of recording the total number of successes is to determine the progress and the basis for displaying the progress bar.

[0140] In this embodiment, the front-end polling query is to query the total number and number of operations of the current task number through the task number. A user can have multiple task numbers, and the progress information of multiple task numbers of the current user will be displayed in the monitoring list at the same time. The progress bar is displayed by providing real-time number of operations and total data. The amount of successful data is the amount of data successfully exported. It is the number of entries in all sheet pages. Success means that all queried data is exported to Excel. The total amount is the number of entries in the source data table queried by the query parameters. Multithreading speeds up processing, and the reduction of memory is to use EasyExcel's stream processing, which can read, process and release at the same time, greatly saving memory consumption. Asynchronous means monitoring the progress bar through Redis while importing / exporting data. Polling scheduling is to read Redis data to generate a progress bar.

[0141] In this embodiment, thread pool is used for thread initialization. The number of threads is the maximum number of threads set by the thread pool. The basis of this number is the number of cores of the application server CPU. The more cores there are, the more threads can be opened. The default is 10, which can be set flexibly. Using multithreading will not directly reduce resource consumption. It is necessary to cooperate with the stream processing of EasyExcel to allow memory data to be used and released at any time, so as to reduce memory consumption. The multithreading of the prior art only improves efficiency, but in the case of millions of data volumes, using only multithreading cannot solve the problem of memory overflow. The verification area is configured correspondingly with the form data and the database field. For example, Table A has three fields a, b, and c, and the field lengths are 1, 10, and 20 respectively, where b is a key field that needs to be filled in. At this time, the properties of its fields are obtained by associating the newly created form data to Table A. When data is imported into Table A, a verification area will be automatically added before the data row, and the imported data can be verified by the general verification function.

[0142] Embodiment 3

[0143] Based on the above embodiment, a data export device of an information system is provided, which includes:

[0144] An export request module, used for receiving an export request;

[0145] An export data volume module, used to respond to the export request, parse the export request, and obtain the export data volume of the data to be exported;

[0146] An export volume detection module is used to detect whether the export data volume is greater than a preset data volume threshold;

[0147] A query writing module, used for, when the amount of exported data is greater than the preset data amount threshold, splitting the data to be exported into first sub-data of a preset number of groups, using a preset number of threads to query the first sub-data of a preset number of groups in the source database respectively, and writing the first sub-data obtained by the query into a preset number of memory pages respectively to obtain the first page data of a preset number of groups, wherein each thread queries a group of the first sub-data, and each thread writes the first page data obtained by the query into a memory page;

[0148] The data merging module is used to merge the first page data to obtain exported data.

[0149] In one embodiment, the query writing module includes:

[0150] A first divided data query and write module is used to write the first divided data obtained by query into a preset number of memory pages respectively;

[0151] A successful export amount detection module is used to detect the amount of data of the first data segment written to the memory page to obtain the successful data amount;

[0152] A successful export amount storage module is used to store the successful data amount in a memory database;

[0153] The device also includes:

[0154] An export quantity storage module, used for storing the export data quantity in the memory database;

[0155] An export amount reading module, configured to read the successful data amount and the export data amount from the memory database in response to receiving a progress query request;

[0156] The export progress calculation module is used to calculate the ratio of the successful data volume to the exported data volume to obtain the export progress.

[0157] In one embodiment, the device further comprises:

[0158] An export empty data detection module, used for detecting whether the data to be exported is empty data when the amount of the exported data is less than or equal to the preset data amount threshold;

[0159] A blank data output module, used for outputting the data to be exported when the data to be exported is blank data;

[0160] The direct export module is used to write the data to be exported into the memory page when the data to be exported is not empty data, so as to obtain the exported data.

[0161] In one embodiment, the query writing module is used to query the first data of a preset number of groups in the source database based on the stream processing component using a preset number of threads, and write the data obtained by the query into a preset number of memory pages to obtain the first page data of a preset number of groups.

[0162] In one embodiment, the device comprises:

[0163] An import request module, used for receiving an import request;

[0164] An import data volume module, used to respond to the import request, parse the import request, and obtain the import data volume of the data to be imported;

[0165] An import volume detection module is used to detect whether the import data volume is greater than a preset data volume threshold;

[0166] An import query and write module, used for, when the amount of imported data is greater than the preset data amount threshold, splitting the data to be imported into a preset number of groups of second sub-data, using a preset number of threads to query the preset number of groups of second sub-data in the source document respectively, and writing the second sub-data obtained by the query into a preset number of memory pages respectively to obtain a preset number of groups of second page data, wherein each thread queries a group of the second sub-data, and each thread writes the second page data obtained by the query into a memory page;

[0167] The import merging module is used to merge the second page data to obtain imported data, and store the imported data in the source database.

[0168] In one embodiment, the device comprises:

[0169] A source document acquisition module is used to acquire a source document and determine a corresponding form configuration according to the storage requirements of the business data of the source document; wherein the form configuration includes configuration parameters and a corresponding configuration format;

[0170] A verification area determination module, used for traversing the source document and determining the area where each data corresponding to each of the configuration parameters is located in the source document as the verification area;

[0171] A verification area detection module is used to detect whether the data in the verification area conforms to the corresponding configuration format;

[0172] A verification success module is used to store the corresponding data in the source document into the source database when the data in the verification area conforms to the corresponding configuration format;

[0173] The verification exception module is used to store the corresponding data in the source document and the exception information generated by the verification in the object storage when the data in the verification area does not conform to the corresponding configuration format.

[0174] In one embodiment, the configuration format includes that the data format is not empty and / or the data length is within a preset range.

[0175] Embodiment 4

[0176] On the basis of the above embodiments, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above embodiments.

[0177] In some implementations of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0178] In some implementations of this embodiment, a computer program product is provided, including a computer program / instructions, characterized in that when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0179] The processor may include, but is not limited to, one or more processors or microprocessors. Each processor may be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the method in the above embodiments.

[0180] Computer-readable storage media can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Computer-readable storage media may include, but are not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (such as hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0181] The computer-readable storage medium may also store at least one computer executable program / instruction, which may be, for example, a computer-readable instruction. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The computer-readable storage medium may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0182] In addition, the computer device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0183] The processor may communicate with external devices via an I / O bus via a wired or wireless network.

[0184] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0185] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0186] It should be noted that in the present disclosure, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element limited by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0187] Although the embodiments disclosed in the present disclosure are as above, the above contents are only embodiments adopted for facilitating the understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the technical field to which the present disclosure belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. A data export method for an information system, characterized in that: include: receiving an export request; Responding to the export request, parsing the export request, and obtaining the export data volume of the data to be exported; Detecting whether the amount of exported data is greater than a preset data amount threshold; When the amount of exported data is greater than the preset data amount threshold, the data to be exported is divided into a preset number of groups of first sub-data, a preset number of threads are used to query the preset number of groups of first sub-data in the source database respectively, and the first sub-data obtained by the query are written into a preset number of memory pages respectively to obtain a preset number of groups of first page data, wherein each thread queries a group of the first sub-data, and each thread writes the first page data obtained by the query into a memory page; The first page data are merged to obtain the exported data.

2. The method according to claim 1, characterized in that The step of writing the first divided data obtained by query into a preset number of memory pages respectively comprises: Writing the first data obtained from the query into a preset number of memory pages respectively; Detecting the amount of data of the first data segment written to the memory segment to obtain the amount of successful data; Storing the successful data volume in a memory database; The method further comprises: storing the exported data volume in the memory database; In response to receiving a progress query request, reading the successful data volume and the exported data volume from the memory database; The ratio of the successful data volume to the exported data volume is calculated to obtain the export progress.

3. The method according to claim 1, characterized in that The method further comprises: When the amount of exported data is less than or equal to the preset data amount threshold, detecting whether the data to be exported is empty data; When the data to be exported is empty data, outputting the data to be exported; When the data to be exported is not empty data, the data to be exported is written into the memory page to obtain the exported data.

4. The method according to claim 1, characterized in that: The step of using a preset number of threads to query a preset number of groups of first data in the source database respectively, and writing the queryed data into a preset number of memory pages respectively, to obtain the preset number of groups of first page data includes: Based on the stream processing component, a preset number of threads are used to query the first data of a preset number of groups in the source database respectively, and the queried data are written into a preset number of memory pages respectively to obtain the first page data of the preset number of groups.

5. The method according to claim 1, characterized in that The method further comprises: receiving an import request; Responding to the import request, parsing the import request, and obtaining the import data volume of the data to be imported; Detecting whether the amount of imported data is greater than a preset data amount threshold; When the amount of imported data is greater than the preset data amount threshold, the data to be imported is divided into a preset number of groups of second sub-data, a preset number of threads are used to query the preset number of groups of second sub-data in the source document respectively, and the second sub-data obtained by the query are written into a preset number of memory pages respectively to obtain a preset number of groups of second page data, wherein each thread queries a group of the second sub-data, and each thread writes the second page data obtained by the query into a memory page; The second page data are merged to obtain imported data, and the imported data is stored in the source database.

6. The method according to claim 1, characterized in that The method comprises: Obtaining a source document, and determining a corresponding form configuration according to the storage requirements of the business data of the source document; wherein the form configuration includes configuration parameters and a corresponding configuration format; Traversing the source document, determining the area where each data corresponding to each of the configuration parameters is located in the source document as the verification area; Check whether the data in the verification area conforms to the corresponding configuration format; When the data in the verification area conforms to the corresponding configuration format, the corresponding data in the source document is stored in the source database; When the data in the verification area does not conform to the corresponding configuration format, the corresponding data in the source document and the exception information generated by the verification are stored in the object storage.

7. The method according to claim 6, characterized in that The configuration format includes that the data format is not empty and / or the data length is within a preset range.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images