Data exporting method, system and equipment based on dynamic batching and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a data exporting method, system and device based on dynamic batching and a storage medium, and the method comprises the following steps: receiving a retrieval request, and determining metadata information matched with the retrieval request; monitoring a memory vacancy rate, and dynamically updating a data volume threshold value based on the memory vacancy rate; evaluating a data volume based on the metadata information, and if the data volume exceeds the data volume threshold, dividing an entity class data export task based on the metadata information into a plurality of sub-tasks; and executing the subtasks one by one to export all entity class data corresponding to the metadata information in batches. According to the method, the memory is dynamically limited, and the data volume processed at a time is limited based on the memory allocation amount, so that the memory resources are fully utilized, and meanwhile, the memory can be prevented from being impacted, and normal operation of other services can be prevented from being influenced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and specifically to a data export method, system, device and storage medium based on dynamic batching. Background Art

[0002] Excel export function is usually provided through web applications, so this technical field also involves web application development. Web application development includes using HTTP protocol, RESTful API design, web server (such as Tomcat, Jetty) and other technologies to build applications that users can access through browsers, and using easypoi technology through the Java backend to export data by category and page.

[0003] Easypoi is based on the Apache POI library, which is a popular open source library for processing Microsoft Office documents, especially Excel files.

[0004] However, when faced with large amounts of data or complex Excel files, the Apache POI library faces severe challenges in memory consumption. Since they usually load the entire worksheet into memory at one time, memory consumption increases dramatically when processing large files, causing a shock to the memory. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a data export method, system, device and storage medium based on dynamic batching to solve the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a data export method based on dynamic batching, comprising: receiving a search request and determining metadata information matching the search request; Monitor the memory idle rate and dynamically update the data volume threshold based on the memory idle rate; Evaluate the data volume based on the metadata information, and if the data volume exceeds the data volume threshold, divide the entity class data export task based on the metadata information into multiple subtasks; The subtasks are executed one by one to export all entity class data corresponding to the metadata information in batches.

[0007] In an optional implementation, a search request is received, and metadata information matching the search request is determined, wherein the metadata information includes index information of target data, including: Extracting a query condition from the search request, searching a database for data related to the query condition, and calculating a similarity between the related data and the query condition; The metadata information of the data whose similarity reaches a first similarity threshold is recorded as the first metadata information, and the metadata information of the data whose similarity is lower than the first similarity threshold and greater than a second similarity threshold is recorded as the second metadata information, wherein the first similarity threshold and the second similarity threshold are pre-set fixed values, and the first similarity threshold is higher than the second similarity threshold.

[0008] In an optional implementation, monitoring the memory idle rate and dynamically updating the data volume threshold based on the memory idle rate includes: Presetting a memory allocation coefficient, wherein the memory allocation coefficient is greater than 0 and less than 1; The product of the memory rated capacity, the memory idle rate and the memory allocation coefficient is used as the size of the dedicated memory area for the data export task; Confirming that the fluctuation value of the memory idle rate is greater than a set fluctuation threshold, dynamically adjusting the dedicated memory area based on the size of the dedicated memory area; Based on the current size of the dedicated memory area, the corresponding data volume threshold is synchronously updated.

[0009] In an optional implementation, the data volume is evaluated based on the metadata information, and if the data volume exceeds the data volume threshold, the entity class data export task based on the metadata information is divided into a plurality of subtasks, including: integrating the first metadata information and the second metadata information into overall metadata information; Evaluate the data volume of the total metadata information and determine whether the data volume of the total metadata information reaches the data volume threshold: If so, then evaluate whether the data volume of the first metadata information exceeds the data volume threshold; if the data volume of the first metadata information does not exceed the data volume threshold, then generate a corresponding data export task based on the first metadata information, and do not perform task splitting; if the data volume of the first metadata information exceeds the data volume threshold, then perform task splitting of the data export task of the first metadata information; If not, a corresponding data export task is generated based on the total metadata information, and task splitting is not performed.

[0010] In an optional implementation, the method for evaluating the data volume based on the metadata information includes: The CARDINALITY function is used to evaluate the number of rows of target data of the retrieval request based on the number of distinct values ​​of the index in the metadata information.

[0011] In an optional implementation, executing the subtasks one by one to export all entity class data corresponding to the metadata information in batches includes: Use easypoi's @Excel annotation to mark the fields of the subtask that need to be exported; In the service layer, according to the paging parameters (such as page number and number of entries per page) passed in by the front end, use the Pageable interface of Spring DataJPA or the paging plug-in of MyBatis Plus to perform paging queries; Traverse the paginated query results, create an Excel Sheet for each page of data, and use the ExcelExportUtil of easypoi or the API of Apache POI to write each page of data into the corresponding Sheet; By setting the HTTP response type, the generated Excel file is written to the output stream of the HTTP response, and the matching response header is set to trigger the file download; Confirm that the subtask is completed and execute the remaining subtasks.

[0012] In a second aspect, the present invention provides a data export system based on dynamic batching, comprising: A request processing module, used for receiving a search request and determining metadata information matching the search request; A threshold updating module, used to monitor the memory idle rate and dynamically update the data volume threshold based on the memory idle rate; A task splitting module, used for evaluating the data volume based on the metadata information, and if the data volume exceeds the data volume threshold, dividing the entity class data export task based on the metadata information into a plurality of subtasks; The task execution unit is used to execute the subtasks one by one to export all entity class data corresponding to the metadata information in batches.

[0013] In an optional implementation, the request processing module includes: A request parsing unit, used to extract a query condition from the search request, query data related to the query condition from a database, and calculate a similarity between the related data and the query condition; The similarity division unit is used to record the metadata information of the data whose similarity reaches a first similarity threshold as the first metadata information, and record the metadata information of the data whose similarity is lower than the first similarity threshold and greater than a second similarity threshold as the second metadata information, wherein the first similarity threshold and the second similarity threshold are pre-set fixed values, and the first similarity threshold is higher than the second similarity threshold.

[0014] In a third aspect, a device is provided, including: A memory for storing a data export program based on dynamic batching; A processor is used to implement the steps of the dynamic batch-based data export method provided in the first aspect when executing the dynamic batch-based data export program.

[0015] In a fourth aspect, a computer-readable storage medium is provided, on which a data export program based on dynamic batching is stored. When the data export program based on dynamic batching is executed by a processor, the steps of the data export method based on dynamic batching provided in the first aspect are implemented.

[0016] The beneficial effect of the present invention lies in that the data export method, system, device and storage medium based on dynamic batching provided by the present invention dynamically limit the memory and limit the amount of data processed at a single time based on the memory allocation amount. While making full use of memory resources, it can avoid impacting the memory and affecting the normal operation of other businesses.

[0017] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0020] Figure 2 is a schematic block diagram of a system according to an embodiment of the present invention.

[0021] Figure 3 A schematic diagram of the structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0024] The data export method based on dynamic batching provided in the embodiment of the present invention is executed by a computer device, and accordingly, the data export system based on dynamic batching runs in the computer device.

[0025] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject may be a data export system based on dynamic batching. According to different requirements, the order of the steps in the flow chart may be changed, and some may be omitted.

[0026] like Figure 1 As shown, the method includes: S1. Receive a search request and determine metadata information matching the search request.

[0027] First, the system receives a search request from the user. The request may contain specific query conditions, screening criteria, or the type of data to be retrieved. Based on the request information, the system searches for matching metadata information in the internal database. Metadata usually contains descriptive information about the data, such as data structure, data type, data relationship, etc. It helps the system understand the meaning and location of the data. By matching the search request with the metadata, the system can determine which data needs to be retrieved and exported.

[0028] S2. Monitor the memory idle rate and dynamically update the data volume threshold based on the memory idle rate.

[0029] Monitor the current memory idle rate in real time, that is, the proportion of unused memory space. The memory idle rate is one of the important indicators of system performance, which affects the data processing speed and efficiency. The system dynamically adjusts the data volume threshold according to the current memory idle rate. When the memory idle rate is high, the system can increase the data volume threshold to allow more data to be processed at one time; when the memory idle rate is low, the system lowers the data volume threshold to avoid memory overflow or performance degradation. This dynamic adjustment strategy helps the system strike a balance between efficiency and stability.

[0030] S3. Evaluate the data volume based on the metadata information, and if the data volume exceeds the data volume threshold, divide the entity class data export task based on the metadata information into multiple subtasks.

[0031] The amount of data to be exported is evaluated based on the previously determined metadata information. If the estimated amount of data exceeds the currently set data volume threshold, the system will automatically split the original export task into multiple smaller subtasks. Each subtask contains part of the data to be exported, and each subtask can be executed independently. This splitting strategy helps to reduce the amount of data exported at a time, thereby reducing the burden on memory and CPU, and improving the system's processing power and response speed.

[0032] S4. Execute the subtasks one by one to export all entity class data corresponding to the metadata information in batches.

[0033] Execute each subtask one by one in the order of the previously divided subtasks. Each subtask will export the corresponding entity class data to the specified location or format according to the specified export field range and the default value of the export template table item. In this way, the system can export a large amount of data in batches, avoiding memory overflow or performance degradation caused by exporting too much data at one time. At the same time, the system can also monitor and record the execution of each subtask so that it can be checked and handled in time when problems arise.

[0034] In an embodiment of the present invention, based on step S1, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0035] Extracting query conditions from the retrieval request, querying data related to the query conditions from a database, and calculating similarity between the related data and the query conditions; recording metadata information of data whose similarity reaches a first similarity threshold as first metadata information, and recording metadata information of data whose similarity is lower than the first similarity threshold and greater than a second similarity threshold as second metadata information, wherein the first similarity threshold and the second similarity threshold are pre-set fixed values, and the first similarity threshold is higher than the second similarity threshold.

[0036] In a specific example, assume that according to a user's search request, relevant books are retrieved from a database and sorted and classified according to the similarity between the books and the search conditions.

[0037] Retrieval request: The user enters the search request: "beginner programming books".

[0038] Step 1: Extract query conditions From the user's search request, the query conditions are extracted: "programming" and "getting started".

[0039] Step 2: Query relevant data According to the extracted query conditions, a query operation is performed in the database to find all book records containing the two keywords "programming" and "beginner".

[0040] Step 3: Calculate similarity For each book found, calculate the similarity between its title, description, keywords and the user's query. The similarity calculation may be based on a variety of algorithms, such as TF-IDF (term frequency-inverse document frequency), cosine similarity, etc. In this example, it is assumed that an effective similarity calculation mechanism has been implemented.

[0041] Step 4: Categorize metadata information Two similarity thresholds were set: the first similarity threshold was 0.8 (representing a high degree of correlation), and the second similarity threshold was 0.5 (representing a certain degree of correlation).

[0042] For books whose similarity reaches or exceeds 0.8, their metadata information (such as title, author, publisher, ISBN, etc.) is recorded as the first metadata information.

[0043] For books with similarity between 0.5 and 0.8, their metadata information is recorded as second metadata information. Although these books have a certain relevance to the user's search request, they may not be as direct or accurate as the books corresponding to the first metadata information.

[0044] For books with similarity lower than 0.5, choose to ignore them.

[0045] In an embodiment of the present invention, based on step S2, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0046] The specific process of allocating memory resources for data export tasks and determining the data volume threshold based on memory resources is as follows: S201. Pre-set memory allocation coefficient During the system design and implementation phase, a memory allocation coefficient α is preset, which is a value between 0 and 1. For example, α can be set to 0.6, which means that 60% of the total system memory is willing to be allocated as a dedicated memory area for the data export task.

[0047] S202. Calculate the size of the dedicated memory area When the system is running, it continuously monitors the rated capacity M (i.e., the total system memory size) and the memory idle rate R (i.e., the proportion of memory currently not in use). Then, the size M_dedicated of the dedicated memory area for the data export task is calculated according to the following formula: M_dedicated = M * R * α; This formula takes into account the actual free memory situation and the proportion of memory willing to be allocated for data export tasks.

[0048] S203. Monitor the fluctuation of memory idle rate In addition to continuously monitoring the memory idle rate R, its fluctuation value ΔR is also calculated, that is, the change in the memory idle rate between two adjacent monitoring time points. If ΔR is greater than the pre-set fluctuation threshold ΔR_threshold, it indicates that the memory idle rate has changed significantly and the size of the dedicated memory area may need to be adjusted.

[0049] S204. Dynamically adjust the dedicated memory area When it is confirmed that the fluctuation value of the memory idle rate is greater than the set fluctuation threshold, the size of the dedicated memory area M_dedicated is recalculated according to the current memory idle rate R and the memory allocation coefficient α, and the memory allocation is adjusted accordingly. Memory size adjustment meets the needs of data export tasks by releasing some no longer needed memory resources or borrowing some memory resources from other tasks.

[0050] S205. Synchronous update data volume threshold After adjusting the size of the dedicated memory area, you also need to update the data volume threshold T related to the dedicated memory area size. This threshold determines the maximum amount of data that the system can process or export at one time under given memory resources. The data volume threshold T can be calculated based on the size of the dedicated memory area M_dedicated, the memory size required for each data item m_per_item, and a safety factor β (usually less than 1 to ensure the safety of memory use): T = M_dedicated / (m_per_item * β); This formula ensures that the data export task can run stably under given memory resources, while leaving a certain safety margin to deal with unforeseen situations.

[0051] Assume that the rated memory capacity M of the system is 16GB, the initial memory idle rate R is 70%, the memory allocation coefficient α is 0.6, the fluctuation threshold ΔR_threshold is 10%, the memory size m_per_item required for each data item is 1MB, and the safety factor β is 0.9.

[0052] Initially, the size of the dedicated memory area M_dedicated is 16GB * 70% * 0.6 = 6.72GB.

[0053] The data volume threshold T is 6.72GB / (1MB * 0.9) ≈ 7466 data items.

[0054] If the memory free rate R suddenly drops to 50% (the fluctuation value is greater than 10%), the size of the dedicated memory area is recalculated to 16GB * 50% * 0.6 = 4.8GB, and the data volume threshold T is updated accordingly to 4.8GB / (1MB * 0.9) ≈ 5333 data items.

[0055] In an embodiment of the present invention, based on step S3, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0056] Among them, the method of evaluating the data volume based on metadata information includes: Assume there is a table called employees with the following fields: employee_id (employee ID, primary key) department_id (department ID, foreign key) first_name last_name salary Also, an index is created on the department_id field.

[0057] Sample query Now, you want to query all employees in a specific department (for example, department ID 10), and want to use the CARDINALITY of the index (or a similar concept) to estimate the number of rows that meet the condition.

[0058] Step 1: Understand the concept of cardinality Ideally, CARDINALITY would tell the number of distinct values ​​in the index. In this example, it might tell how many distinct department IDs there are in the department_id index. In practice, however, database systems don't usually directly provide a property called CARDINALITY to query this; instead, they might provide statistics or metadata from which similar information can be inferred.

[0059] Step 2: Query statistics In most database systems, query system views or tables to obtain index statistics. For example, in Oracle, query the DBA_INDEXES and DBA_TAB_STATISTICS views; in MySQL, use the SHOW INDEX statement and query the INFORMATION_SCHEMA.STATISTICS table; in PostgreSQL, query the pg_stat_all_indexes and pg_stats views.

[0060] However, to simplify this example, we will assume there is a fictional function called index_cardinality that returns the number of distinct values ​​in an index.

[0061] Step 3: Use statistics to estimate the number of rows If you know the CARDINALITY of the department_id index (i.e. the number of distinct department IDs), and you know that the specific department IDs you are querying for (say 10) are unique (or nearly unique, i.e. there are very few duplicates) within the index, then you can roughly estimate the number of rows that meet your condition.

[0062] Suppose index_cardinality('employees_department_id_idx') returns 100 (indicating that there are 100 different department IDs), and we know that the department with ID 10 exists in the index.

[0063] Step 4: Build the query and explain Based on the above assumptions, one might construct a query like the following (note, however, that this query is fictional, as there is no actual function called index_cardinality): SELECT COUNT(*) FROM employees WHERE department_id = 10; Since we know that the CARDINALITY of the department_id index is 100, and we assume that the department with department ID 10 is unique in the index, we roughly estimate the number of rows that meet the condition to be close to 1.

[0064] It can be understood that in one embodiment of the present invention, in order to output valid data preferentially, a step-by-step processing method is adopted, specifically including: Evaluate the data volume of the total metadata information and determine whether the data volume of the total metadata information reaches the data volume threshold: If so, then evaluate whether the data volume of the first metadata information exceeds the data volume threshold; if the data volume of the first metadata information does not exceed the data volume threshold, then generate a corresponding data export task based on the first metadata information, and do not perform task splitting; if the data volume of the first metadata information exceeds the data volume threshold, then perform task splitting of the data export task of the first metadata information; If not, a corresponding data export task is generated based on the total metadata information, and task splitting is not performed.

[0065] Among them, generating the corresponding data export task based on the metadata information refers to generating entity class data for the database data indicated by the metadata information. Specifically, the method is as follows: (1) Specify the database connection information in the ORM framework configuration file, including the database URL, user name, password, etc.

[0066] (2) Use annotations to specify a class as the base class of the dynamic entity class, and fill in specific properties through reflection or dynamic proxy at runtime.

[0067] (3) Parse the table name and field information of the corresponding data based on the metadata information.

[0068] (4) Dynamically generate the corresponding entity class based on the parsed table name and field information. This can be achieved through reflection mechanism, bytecode operation library (such as ASM, Javassist) or API provided by ORM framework.

[0069] (5) Use the query API provided by the ORM framework, set the dynamically generated entity class as the target type of the query result, execute the SQL query and obtain the results.

[0070] In an embodiment of the present invention, based on step S4, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0071] In a specific example, the execution flow of the subtasks is as follows: S401. Define data model: Use easypoi's @Excel annotation to mark the fields of the subtask that need to be exported.

[0072] S402. Implement paging query: In the service layer, based on the paging parameters (such as page number and number of entries per page) passed in by the front end, the data layer is called to perform paging queries.

[0073] Use Spring Data JPA's Pageable interface or MyBatis Plus's paging plugin to simplify paging logic.

[0074] S403. Build Excel export logic: Traverse the paginated query results and create an Excel Sheet for each page of data.

[0075] If easypoi directly supports multi-sheet export, configure it according to its API documentation.

[0076] If easypoi does not support exporting multiple sheets directly, you can use Apache POI's Workbook and Sheet API to manually create and add sheets.

[0077] Use easypoi's ExcelExportUtil or Apache POI's API to write each page of data into the corresponding Sheet.

[0078] S404. Set HTTP response: In the control layer, set the Content-Type of the HTTP response to application / vnd.ms-excel.

[0079] Write the generated Excel file to the output stream of the HTTP response and set the appropriate response headers (such as Content-Disposition) to trigger the file download.

[0080] Confirm that the current subtask is completed, and continue to execute other subtasks until all subtasks are completed. When all subtasks are completed, use the Excel tool to merge the corresponding multiple Excel tables.

[0081] In some embodiments, the data export system based on dynamic batching may include multiple functional modules composed of computer program segments. The computer program of each program segment in the data export system based on dynamic batching may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Function of data export based on dynamic batching.

[0082] In this embodiment, the data export system based on dynamic batching can be divided into multiple functional modules according to the functions it performs, such as Figure 2 As shown. The functional modules of the system 200 may include: a request processing module 210, a threshold updating module 220, a task splitting module 230 and a task execution module 240. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0083] A request processing module, used for receiving a search request and determining metadata information matching the search request; A threshold updating module, used to monitor the memory idle rate and dynamically update the data volume threshold based on the memory idle rate; A task splitting module, used for evaluating the data volume based on the metadata information, and if the data volume exceeds the data volume threshold, dividing the entity class data export task based on the metadata information into a plurality of subtasks; The task execution unit is used to execute the subtasks one by one to export all entity class data corresponding to the metadata information in batches.

[0084] Optionally, as an embodiment of the present invention, the request processing module includes: A request parsing unit, used to extract a query condition from the search request, query data related to the query condition from a database, and calculate a similarity between the related data and the query condition; The similarity division unit is used to record the metadata information of the data whose similarity reaches a first similarity threshold as the first metadata information, and record the metadata information of the data whose similarity is lower than the first similarity threshold and greater than a second similarity threshold as the second metadata information, wherein the first similarity threshold and the second similarity threshold are pre-set fixed values, and the first similarity threshold is higher than the second similarity threshold.

[0085] Optionally, as an embodiment of the present invention, dynamically updating the data volume threshold based on the memory idle rate includes: Presetting a memory allocation coefficient, wherein the memory allocation coefficient is greater than 0 and less than 1; The product of the memory rated capacity, the memory idle rate and the memory allocation coefficient is used as the size of the dedicated memory area for the data export task; Confirming that the fluctuation value of the memory idle rate is greater than a set fluctuation threshold, dynamically adjusting the dedicated memory area based on the size of the dedicated memory area; Based on the current size of the dedicated memory area, the corresponding data volume threshold is synchronously updated.

[0086] Optionally, as an embodiment of the present invention, the data volume is evaluated based on the metadata information, and if the data volume exceeds the data volume threshold, the entity class data export task based on the metadata information is divided into multiple subtasks, including: integrating the first metadata information and the second metadata information into overall metadata information; Evaluate the data volume of the total metadata information and determine whether the data volume of the total metadata information reaches the data volume threshold: If so, then evaluate whether the data volume of the first metadata information exceeds the data volume threshold; if the data volume of the first metadata information does not exceed the data volume threshold, then generate a corresponding data export task based on the first metadata information, and do not perform task splitting; if the data volume of the first metadata information exceeds the data volume threshold, then perform task splitting of the data export task of the first metadata information; If not, a corresponding data export task is generated based on the total metadata information, and task splitting is not performed.

[0087] Optionally, as an embodiment of the present invention, a method for evaluating data volume based on the metadata information includes: The CARDINALITY function is used to evaluate the number of rows of target data of the retrieval request based on the number of distinct values ​​of the index in the metadata information.

[0088] Optionally, as an embodiment of the present invention, executing the subtasks one by one to export all entity class data corresponding to the metadata information in batches includes: Use easypoi's @Excel annotation to mark the fields of the subtask that need to be exported; In the service layer, according to the paging parameters (such as page number and number of entries per page) passed in by the front end, use the Pageable interface of Spring DataJPA or the paging plug-in of MyBatis Plus to perform paging queries; Traverse the paginated query results, create an Excel Sheet for each page of data, and use the ExcelExportUtil of easypoi or the API of Apache POI to write each page of data into the corresponding Sheet; By setting the HTTP response type, the generated Excel file is written to the output stream of the HTTP response, and the matching response header is set to trigger the file download; Confirm that the subtask is completed and execute the remaining subtasks.

[0089] Figure 3The data export method based on dynamic batching provided for the embodiment of the present application can be applied to equipment. It will be appreciated by those skilled in the art that the equipment structure involved in the embodiment of the present invention does not constitute a limitation on the equipment, and the equipment may include more or less components than shown, or combine certain components, or different component arrangements. In an embodiment of the present invention, the equipment includes but is not limited to laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The equipment may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the embodiments of the present application described herein and / or required.

[0090] The device 300 may include: a processor 310, a memory 320 and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0091] The memory 320 may be used to store the execution instructions of the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is enabled to perform some or all of the steps in the following method embodiments.

[0092] The processor 310 is the control center of the storage device, and uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 310 can include only a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0093] The communication unit 330 is used to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0094] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0095] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0096] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0097] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0098] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0100] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A data export method based on dynamic batching, characterized in that: include: receiving a search request and determining metadata information matching the search request; Monitor the memory idle rate and dynamically update the data volume threshold based on the memory idle rate; Evaluate the data volume based on the metadata information, and if the data volume exceeds the data volume threshold, divide the entity class data export task based on the metadata information into multiple subtasks; The subtasks are executed one by one to export all entity class data corresponding to the metadata information in batches.

2. The method according to claim 1, characterized in that Receive a search request, and determine metadata information matching the search request, the metadata information including index information of target data, including: Extracting a query condition from the search request, searching a database for data related to the query condition, and calculating a similarity between the related data and the query condition; The metadata information of the data whose similarity reaches a first similarity threshold is recorded as the first metadata information, and the metadata information of the data whose similarity is lower than the first similarity threshold and greater than a second similarity threshold is recorded as the second metadata information, wherein the first similarity threshold and the second similarity threshold are pre-set fixed values, and the first similarity threshold is higher than the second similarity threshold.

3. The method according to claim 1, characterized in that Monitoring the memory idle rate and dynamically updating the data volume threshold based on the memory idle rate, including: Presetting a memory allocation coefficient, wherein the memory allocation coefficient is greater than 0 and less than 1; The product of the memory rated capacity, the memory idle rate and the memory allocation coefficient is used as the size of the dedicated memory area for the data export task; Confirming that the fluctuation value of the memory idle rate is greater than a set fluctuation threshold, dynamically adjusting the dedicated memory area based on the size of the dedicated memory area; Based on the current size of the dedicated memory area, the corresponding data volume threshold is synchronously updated.

4. The method according to claim 2, characterized in that: The data volume is evaluated based on the metadata information. If the data volume exceeds the data volume threshold, the entity class data export task based on the metadata information is divided into a plurality of subtasks, including: integrating the first metadata information and the second metadata information into overall metadata information; Evaluate the data volume of the total metadata information and determine whether the data volume of the total metadata information reaches the data volume threshold: If so, then evaluate whether the data volume of the first metadata information exceeds the data volume threshold; if the data volume of the first metadata information does not exceed the data volume threshold, then generate a corresponding data export task based on the first metadata information, and do not perform task splitting; if the data volume of the first metadata information exceeds the data volume threshold, then perform task splitting of the data export task of the first metadata information; If not, a corresponding data export task is generated based on the total metadata information, and task splitting is not performed.

5. The method according to claim 1, characterized in that The method for evaluating the data volume based on the metadata information includes: The CARDINALITY function is used to evaluate the number of rows of target data of the retrieval request based on the number of distinct values ​​of the index in the metadata information.

6. The method according to claim 1, characterized in that Executing the subtasks one by one to export all entity class data corresponding to the metadata information in batches includes: Use easypoi's @Excel annotation to mark the fields of the subtask that need to be exported; In the service layer, use the Pageable interface of Spring Data JPA or the paging plug-in of MyBatis Plus to perform paging queries based on the paging parameters (such as page number and number of entries per page) passed in by the front end; Traverse the paginated query results, create an Excel Sheet for each page of data, and use the ExcelExportUtil of easypoi or the API of Apache POI to write each page of data into the corresponding Sheet; By setting the HTTP response type, the generated Excel file is written to the output stream of the HTTP response, and the matching response header is set to trigger the file download; Confirm that the subtask is completed and execute the remaining subtasks.

7. A data export system based on dynamic batching, characterized in that: include: A request processing module, configured to receive a search request and determine metadata information matching the search request; A threshold updating module, used to monitor the memory idle rate and dynamically update the data volume threshold based on the memory idle rate; A task splitting module, used for evaluating the data volume based on the metadata information, and if the data volume exceeds the data volume threshold, dividing the entity class data export task based on the metadata information into a plurality of subtasks; Parameter adjustment module, used to adjust the export field range and the default value of the export template table item; The task execution module is used to execute the subtasks one by one to export all entity class data corresponding to the metadata information in batches.

8. The system according to claim 7, characterized in that The request processing module includes: A request parsing unit, used to extract a query condition from the search request, query data related to the query condition from a database, and calculate a similarity between the related data and the query condition; The similarity division unit is used to record the metadata information of the data whose similarity reaches a first similarity threshold as the first metadata information, and record the metadata information of the data whose similarity is lower than the first similarity threshold and greater than a second similarity threshold as the second metadata information, wherein the first similarity threshold and the second similarity threshold are pre-set fixed values, and the first similarity threshold is higher than the second similarity threshold.

9. A device, characterized in that: include: A memory, used for storing a data export program based on dynamic batching; A processor, configured to implement the steps of the dynamic batch-based data export method as described in any one of claims 1 to 6 when executing the dynamic batch-based data export program.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a data export program based on dynamic batching, and when the data export program based on dynamic batching is executed by a processor, the steps of the data export method based on dynamic batching as described in any one of claims 1 to 6 are implemented.

Images