Excel-based large data volume import and export processing method, system and equipment and medium

Abstract

The invention provides a large data volume import and export processing method, system and device based on Excel and a medium, and belongs to the technical field of batch data process.The method comprises the steps that statistics is conducted on user query information of a service system, and a unified query model is constructed; creating an asynchronous data processing component, packaging the asynchronous data processing component based on the unified query model, and adding the packaged asynchronous data processing component to a service system; the asynchronous data processing component responds to an Excel-based data import or export request, analyzes user query information by using a unified query model, determines a target operation and a data flow direction, and generates a target task; and the scheduling service receives the target task, starts an asynchronous operation, divides the target task according to the target task and the server resource condition, and feeds back to the front-end page of the business system until the target task is completed. By constructing the unified query model and the asynchronous data processing component, efficient data processing and reasonable resource scheduling are realized, the cost is reduced, the data security is guaranteed, and the user experience is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of batch data processing, and particularly relates to a method, system, device and medium for importing and exporting large amounts of data based on Excel. Background Art

[0002] With the development of digital business, enterprise business systems often involve a large number of data import and export operations, especially batch data processing based on Excel format. The import and export of traditional business systems mostly adopt real-time data processing methods. In scenarios where the data volume is large or during peak business periods, such as monthly and annual settlements of financial systems, and data statistics after major promotions in e-commerce systems, situations where the concurrency, computing volume, and data volume are all large will occur. This will lead to problems such as server resource exhaustion and system freezing, seriously affecting user experience and service availability. Traditional solutions usually involve increasing application extensions or upgrading server configurations, indirectly increasing the operation and maintenance costs of the project. At the same time, due to the lack of unified planning for different business scenarios, the processing reusability is poor, further increasing the development workload.

[0003] Currently, the real-time data processing method mainly has the following defects: First, in scenarios of large data volume and high concurrency, real-time processing easily leads to a sharp decline in the performance of business systems. Second, users need to wait for a long time for data processing to complete, seriously affecting the user experience. Third, during peak business periods, the business system may be unable to provide normal services due to resource exhaustion. Finally, relying on the method of software and hardware upgrades indirectly increases the operating costs of business systems, and it is difficult to make major breakthroughs in the development of software and hardware due to the need to overcome technical bottlenecks. Summary of the Invention

[0004] In a first aspect, an embodiment of the present application provides a method for importing and exporting large amounts of data based on Excel, including the following steps: S1. Statistically analyze the user query information of the business system and construct a unified query model; S2. Create an asynchronous data processing component, encapsulate it based on the unified query model, and add it to the business system; S3. The asynchronous data processing component responds to an Excel-based data import or export request, uses the unified query model to parse the user query information, determines the target operation and data flow direction, and generates a target task; S4. The scheduling service receives the target task, starts an asynchronous operation, divides the target task according to the target task and server resource conditions until the target task is completed and feeds back to the front-end page of the business system.

[0005] Further, the specific steps of step S1 are as follows: S11. Collect the user query information in the business system; the user query information includes query resources, retrieval conditions, and data permissions. S12. Analyze the collected user query information, construct a unified data structure and unified query logic, and obtain a unified query model.

[0006] Further, the specific steps of step S12 are as follows: S121. Identify the data types of business objects in different scenarios according to the collected user query information, and establish a unified basic data object; the unified basic data object includes general fields and specific business fields representing business scenario attributes. S122. Define a unified retrieval condition data structure to describe the user's query requirements. S123. Establish a data permission data structure to store the user's access permissions to different business objects. S124. Use the unified basic data object, unified retrieval condition data structure, and data permission data structure as the unified data structure. S125. Define the unified query logic as: convert the retrieval conditions in the user query information according to the unified retrieval condition data structure, verify them using the data permission data structure, then execute the query of the unified basic data object to the data source using the converted and verified retrieval conditions, and finally process and return the query results.

[0007] Further, the specific steps of step S2 are as follows: S21. Select open tools and frameworks according to the programming language and operating environment used by the business system. S22. Use the selected development tools and design an asynchronous data processing component based on the selected framework, add a data import module and a data export module, and design a data verification module and a log recording module using tool classes respectively. S23. Bind and encapsulate the asynchronous data processing component with the unified query model, and design an interface with the business system to define the request parameter format as input and the return result format as output. S24. Add the asynchronous data processing component to the business system through the designed interface.

[0008] Further, the specific steps of step S3 are as follows: S31. After the business system receives a data import or export request based on Excel, input it into the asynchronous data processing component according to the request parameter format. S32. The asynchronous data processing component obtains the user query information from the data import or export request, and parses and filters the user query information using the unified query model. S33. The asynchronous data processing component uses the data import module to process the user query information of the data import request, and uses the data export module to process the user query information of the data export request; S34. The asynchronous data processing component uses the data verification module to verify the processing result of the user query information, and determines whether the target operation is data import or data export according to the processing result of the user query information that passes the verification; If the target operation is data import, it is determined that the data flow is to export data from the Excel table to the business system; If the target operation is data export, it is determined that the data flow is to export data from the business system to the Excel table; S35. It is determined that the target operation needs to be executed on the processing result of the user query information to obtain the target task, and send it to the scheduling service.

[0009] Furthermore, the specific steps of step S32 are as follows: S321. The asynchronous data processing component executes the unified query logic; S322. The asynchronous data processing component extracts the retrieval conditions from the user query information and converts them according to the unified retrieval condition data structure; S323. The asynchronous data processing component filters the retrieval conditions using the data permission data structure; S324. The asynchronous data processing component applies the filtered retrieval conditions to the query of the unified basic data object to determine the qualified data.

[0010] Furthermore, the specific steps of step S4 are as follows: S41. The scheduling service receives the target task, stores it in the database, and starts the asynchronous operation of the business system; S42. The scheduling server starts the target task scheduling; S43. The scheduling service obtains the server resource occupancy rate and judges whether the server is resource idle; If so, go to step S44; If not, wait for the set time period and return to step S42; S44. The scheduling server obtains the data volume of the target task and judges whether it exceeds the server's resource load; If so, return to step S42; If not, go to step S45; S45. The scheduling server calculates the number of parallel tasks according to the server's resource situation and the data volume of the target task, and splits the target task into several subtasks according to the number of parallel tasks; S46. The scheduling server allocates the server resources to execute the subtasks, and merges the execution results of each subtask; S47. The scheduling server responds to the merged data to generate a message notification and provides it to the front-end page of the business system, and converts the merged data into a return result format and provides it to the asynchronous data processing component; S48. The asynchronous data processing component returns the merged data to the front-end interface of the business system; S49. The front-end interface of the business system responds to the user's data import or export query, performs operations on the merged data, and provides a data processing status query and a data file download interface.

[0011] In a second aspect, an embodiment of the present application further provides a large data volume import and export processing system based on Excel, including: A unified query model construction module, configured to count the user query information of the business system and construct a unified query model; An asynchronous data processing component creation module, configured to create an asynchronous data processing component, and encapsulate it based on the unified query model and add it to the business system; A target task generation module, configured to respond to an Excel-based data import or export request through the asynchronous data processing component, parse the user query information using the unified query model, determine the target operation and data flow direction, and generate a target task; A target task scheduling module, configured to receive the target task in the scheduling service to start an asynchronous operation, divide the target task according to the target task and the server resource situation until the target task is completed and feedback to the front-end page of the business system.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the large data volume import and export processing method based on Excel as described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present application further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the large data volume import and export processing method based on Excel as described in the first aspect are implemented.

[0014] It can be seen from the above technical solutions that the present invention has the following advantages: In the method, system, device, and medium for processing large amounts of data import and export based on Excel provided by this application, by constructing a unified query model and an asynchronous data processing component, efficient processing of large amounts of data is achieved, reducing the waiting time of users and improving the response speed of the business system. Especially when importing and exporting data, tasks can be completed quickly and accurately; a reasonable resource scheduling and task division mechanism is implemented, avoiding server resource exhaustion and system freeze caused by excessive data volume or high concurrency, and ensuring the stable operation of the business system; through the unified query model and standardized interface design, the business system can smoothly access the asynchronous data processing component, and even achieve zero-code docking in simple scenarios, reducing development and maintenance costs; through the data verification module and strict permission verification mechanism, the accuracy and integrity of the imported and exported data are ensured, while preventing data leakage and illegal access; through the front-end page for task tracking and querying the data processing status, and providing a data file download interface, users can understand the data processing progress in real time and obtain the processing results conveniently and quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 It is a schematic flowchart of the method for processing large amounts of data import and export based on Excel of the present invention.

[0017] Figure 2 It is a schematic diagram of the system for processing large amounts of data import and export based on Excel of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] In the following, the specific steps of the method for processing large amounts of data import and export based on Excel will be described in detail, and various embodiments of the present disclosure will be described more comprehensively. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but the present disclosure should be understood to cover all adjustments, equivalents, and / or alternative solutions that fall within the spirit and scope of the various embodiments of the present disclosure.

[0019] Exemplarily, with the continuous expansion of digital business, enterprise business systems frequently need to perform a large number of data import and export operations, especially for batch data processing tasks involving Excel format. In traditional business systems, data import and export mostly adopt real-time processing methods. However, in scenarios where the data volume is huge or during peak business hours, such as monthly and annual settlements in the financial system, and data statistics after major promotions in the e-commerce system, the system will face challenges such as a sharp increase in concurrent access volume, heavy computing tasks, and a huge amount of data. In this case, server resources are often quickly exhausted, resulting in slow system response or even "frozen" phenomena, seriously damaging the user experience and service availability.

[0020] Traditional coping strategies usually involve increasing application instances or upgrading server hardware configurations to address the issues, but this undoubtedly increases the operation and maintenance costs of the project. At the same time, due to the lack of unified planning and design for data processing in different business scenarios, the reusability of processing logic is relatively low, which further exacerbates the burden of development work.

[0021] There are many drawbacks to the current real-time data processing method: First, in a large data volume and high-concurrency environment, real-time processing will seriously drag down the performance of the business system; second, users need to wait a long time for the data processing results, greatly affecting the usage experience; third, during peak business hours, the business system may be unable to provide normal services due to insufficient resources; finally, relying on software and hardware upgrades to address problems not only increases the operating costs, but also faces technical bottlenecks that are difficult to break through in the development of software and hardware technologies.

[0022] To address the above problems, this embodiment provides a method for processing large data volume import and export based on Excel. By constructing a unified query model, standardizing data processing rules, optimizing query logic, and improving query efficiency, an asynchronous data processing component is created and encapsulated and added to the business system to implement background task processing, avoid blocking the business system, improve the user experience, schedule the use of services to divide tasks according to task and resource situations, make full use of server resources, avoid waste and backlog, and provide timely feedback after processing to improve the user experience.

[0023] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0024] Please refer to Figure 1 The flowchart of the method for processing large data volume import and export based on Excel in a specific embodiment is shown. The method includes the following steps: S1. Statistically analyze the user query information of the business system to construct a unified query model; It should be noted that the unified query model enables data query and processing in different business scenarios to follow the same rules, improving the consistency of data processing; through the statistical analysis of user query information, the unified query model has strong adaptability; S2. Create an asynchronous data processing component, encapsulate it based on the unified query model, and add it to the business system; It should be noted that the asynchronous data processing component can process data import and export tasks in the background, avoiding blocking the normal operation of the business system; the encapsulation of the asynchronous data processing component based on the unified query model enables the component to seamlessly connect with the business system, facilitating integration and expansion; S3. The asynchronous data processing component responds to Excel-based data import or export requests, uses the unified query model to parse user query information, determines the target operation and data flow direction, and generates a target task; It should be noted that using the unified query model to parse user query information ensures the accuracy of data processing and avoids invalid operations; by clarifying the target operation and data flow direction, it provides a direction for subsequent task execution and improves the efficiency of task processing; S4. The scheduling service receives the target task to start an asynchronous operation, divides the target task according to the target task and server resource conditions until the target task is completed and feedback is sent to the front-end page of the business system; It should be noted that through task division and resource scheduling, server resources are fully utilized, the efficiency of task processing is improved, and resource waste and task backlog are avoided; by timely feedback of the task completion status to the front-end page of the business system, users can obtain the processing results in a timely manner, enhancing the user experience.

[0025] In this embodiment, through the asynchronous data processing component, the asynchronous processing of large-volume data import and export is realized, avoiding system blockage caused by synchronous operations, and improving the response speed and processing capacity of the business system; by constructing a unified query model, user query information in different scenarios can be standardized processed, simplifying the logic of data processing and enhancing the scalability of the business system.

[0026] Furthermore, as a refinement and extension of the specific implementation manner of the above embodiment, in order to fully illustrate the specific implementation process in this embodiment, another method for processing large-volume data import and export based on Excel is provided. This method includes the following steps: S1. Statistically analyze the user query information of the business system to construct a unified query model; The specific steps of step S1 are as follows: S11. Collect user query information in the business system; the user query information includes query resources, retrieval conditions, and data permissions; S12. Analyze the collected user query information, construct a unified data structure and unified query logic, and obtain a unified query model; It should be noted that by collecting and analyzing user query information, the unification of data structure and query logic is realized, providing a data basis for the steps of constructing a unified query model; S2. Create an asynchronous data processing component, encapsulate it based on the unified query model, and add it to the business system; the specific steps of step S2 are as follows: S21. Select open tools and frameworks according to the programming language and operating environment used by the business system; S22. Use the selected development tools and design an asynchronous data processing component based on the selected framework, add a data import module and a data export module, and use tool classes to design a data verification module and a log recording module respectively; S23. Bind and encapsulate the asynchronous data processing component with the unified query model, design an interface with the business system, and define the format of the request parameters as input and the format of the return results as output; S24. Add the asynchronous data processing component to the business system through the designed interface; It should be noted that selecting appropriate development tools and frameworks according to the programming language and operating environment used by the business system ensures the compatibility and stability of the asynchronous data processing component; the asynchronous data processing component improves the maintainability and scalability of the component through modular data import, export, verification, and log recording; and the interface definition makes the asynchronous data processing component easy to integrate into the business system, reducing the integration difficulty and cost; S3. The asynchronous data processing component responds to an Excel-based data import or export request, uses the unified query model to parse user query information, determines the target operation and data flow direction, and generates a target task; the specific steps of step S3 are as follows: S31. After the business system receives an Excel-based data import or export request, input it into the asynchronous data processing component according to the request parameter format; S32. The asynchronous data processing component obtains user query information from the data import or export request, and parses and filters the user query information using the unified query model; S33. The asynchronous data processing component uses the data import module to process the user query information of the data import request, and uses the data export module to process the user query information of the data export request; S34. The asynchronous data processing component uses the data verification module to verify the processing result of the user's query information, and determines whether the target operation is data import or data export according to the processing result of the user's query information that passes the verification; If the target operation is data import, determine that the data flow direction is to import from the business system to the Excel table; If the target operation is data export, determine that the data flow direction is to import from the Excel table to the business system; S35. Determine that the target operation needs to be performed on the processing result of the user's query information to obtain the target task, and send it to the scheduling service; It should be noted that through the unified request parameter format and return result format, it is ensured that the asynchronous data processing component can accurately receive and process the requests of the business system; generating the target task and sending it to the scheduling service realizes the asynchronous processing of the task and improves the resource utilization rate; S4. The scheduling service receives the target task to start the asynchronous operation, divides the target task according to the target task and the server resource situation until the target task is completed and feeds back to the front-end page of the business system; the specific steps of step S4 are as follows: S41. The scheduling service receives the target task and stores it in the database, and starts the asynchronous operation of the business system; S42. The scheduling server starts the target task scheduling; S43. The scheduling service obtains the server resource occupancy rate and judges whether the server is resource idle; If so, go to step S44; If not, wait for the set time period and return to step S42; S44. The scheduling server obtains the data volume of the target task and judges whether it exceeds the server's resource load; If so, return to step S42; If not, go to step S45; S45. The scheduling server calculates the number of parallel tasks according to the server's resource situation and the data volume of the target task, and splits the target task into several subtasks according to the number of parallel tasks; S46. The scheduling server allocates server resources to execute the subtasks, and merges the execution results of each subtask; S47. The scheduling server responds to the merged data to generate a message notification for the front-end page of the business system, and converts the merged data into the return result format for the asynchronous data processing component; S48. The asynchronous data processing component returns the merged data to the front-end interface of the business system; S49. The front-end interface of the business system responds to the user's data import or export query, performs operations on the merged data, and provides a data processing status query and a data file download interface; It should be noted that the scheduling service dynamically calculates the number of parallel tasks according to the server resource situation and the data volume of the target task, and splits the target task, achieving efficient utilization of resources and rapid completion of the task; it merges the execution results of each subtask, generates a message notification and provides it to the front-end page of the business system, improving the integrity and timeliness of data processing; while the front-end interface of the business system provides interfaces for querying the data processing status and downloading data files, enhancing the user experience and system usability.

[0027] In an embodiment of the present invention, based on step S12, a possible embodiment will be given below to non-restrictively elaborate on its specific implementation scheme.

[0028] The specific steps of step S12 are as follows: S121. Identify the data types of business objects in different scenarios according to the collected user query information, and establish a unified basic data object; the unified basic data object includes general fields and specific business fields representing business scenario attributes. Exemplarily, business objects in different business scenarios can be sales data, customer data; general fields can include business ID, scenario time or modification time, and specific business fields need to be defined according to the requirements of different business scenarios. For example, the business object of sales data can include fields such as product name, sales quantity, sales unit price, sales amount, sales area, sales time, etc., which are specifically used to describe detailed sales-related information; while the business object of customer data can include fields such as customer name, contact information (phone, email), customer address, customer type (individual / enterprise), registration time, etc., which can comprehensively record customer-related information. S122. Define a unified retrieval condition data structure to describe the user's query requirements. It should be noted that the retrieval condition data structure can describe the user's query requirements. For example, it can adopt the form of key-value pairs, where the key represents the query field, such as customer name, order date, and the value represents the corresponding query condition, such as a specific customer name, a certain date range; at the same time, logical operators, such as AND, OR, NOT, can be introduced into the retrieval condition data structure to combine multiple retrieval conditions, thereby realizing complex query logic. S123. Establish a data permission data structure to store the access permissions of users to different business objects. It should be noted that the data permission data structure adopts the role-permission matrix method to define the read, write, modify and other permission levels for each role to different business objects. Thus, when querying, the query results are filtered and restricted according to the user's role and the permission data structure. S124. Take the unified basic data object, the unified retrieval condition data structure, and the data permission data structure as the unified data structure; S125. Define the unified query logic as follows: convert the retrieval conditions in the user query information according to the unified retrieval condition data structure, verify them using the data permission data structure, then execute the query of the unified basic data object on the data source using the converted and verified retrieval conditions, and finally process and return the query results.

[0029] In an embodiment of the present invention, based on step S32, a possible embodiment will be given below to non - restrictively elaborate on its specific implementation.

[0030] The specific steps of step S32 are as follows: S321. The asynchronous data processing component executes the unified query logic; S322. The asynchronous data processing component obtains the user query information, extracts the retrieval conditions, and converts them according to the unified retrieval condition data structure; S323. The asynchronous data processing component filters the retrieval conditions using the data permission data structure; S324. The asynchronous data processing component applies the filtered retrieval conditions to the query of the unified basic data object to determine the qualified data; Exemplarily, when the user enters a query condition in the business system, the asynchronous data processing component first parses the input and converts it into a format that conforms to the unified retrieval condition data structure; for example, converts information such as the date range selected by the user on the interface and the keywords entered into a key - value pair - form retrieval condition; before executing the query, according to the user's identity information, such as the user name and role, obtain the user's access permission to the current business object from the data permission data structure; then, filter and adjust the retrieval conditions according to the permission information; for example, if the user only has the permission to view the sales data of some regions, then automatically add a region restriction condition when querying to ensure that the query results conform to the user's permissions; apply the parsed and permission - verified retrieval conditions to the corresponding business object data for query; database query languages such as SQL or other data access technologies can be used to obtain qualified data from the database or other data sources according to the unified data structure and retrieval conditions.

[0031] It should be noted that during the query process, for business objects in different scenarios, the same query execution logic is adopted, and only corresponding adjustments are made according to the data structure of the business object and the specific query conditions.

[0032] It should be understood that the sequence numbers of the steps in the above embodiments do not indicate the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0033] As Figure 2 shown, the following is an embodiment of a large - data - volume import / export processing system based on Excel provided by the embodiments of the present disclosure. This system belongs to the same inventive concept as the large - data - volume import / export processing system based on Excel in the above - mentioned embodiments. For the details not described in detail in the embodiments of the large - data - volume import / export processing system based on Excel, reference can be made to the embodiments of the large - data - volume import / export processing system based on Excel.

[0034] The system includes: A unified query model construction module, which is used to count the user query information of the business system and construct a unified query model; An asynchronous data processing component creation module, which is used to create an asynchronous data processing component, encapsulate it based on the unified query model, and add it to the business system; A target task generation module, which is used to respond to an Excel - based data import or export request through the asynchronous data processing component, parse the user query information using the unified query model, determine the target operation and data flow direction, and generate a target task; A target task scheduling module, which is used to start an asynchronous operation when the scheduling service receives the target task, divide the target task according to the target task and server resource conditions, and feed back to the front - end page of the business system until the target task is completed.

[0035] Through the collaborative interaction among the unified query model construction module, the asynchronous data processing component creation module, the target task generation module, and the target task scheduling module in this embodiment, the efficiency of large - data - volume import / export processing based on Excel is improved, and the user experience is enhanced.

[0036] The big data volume import / export processing method provided by the embodiments of this application can be applied to an electronic device. Those skilled in the art can understand that the structure of the electronic device involved in the embodiments of the present invention does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. In the embodiments of the present invention, the electronic device includes, but is not limited to, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0037] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a key, a camera, a display screen, and a SIM card interface, etc.

[0038] It can be understood that the structure schematically shown in the embodiments of this application does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0039] The processor may include one or more processing units. For example, the processor may include a central processing unit (CPU), etc., an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0040] Among them, the processor can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0041] A memory can also be set in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can save the instructions or data that the processor has just used or recycled. If the processor needs to use the instruction or data again, it can directly call it from this memory, avoiding repeated accesses, reducing the waiting time of the processor, and thus improving the system efficiency.

[0042] The above-mentioned electronic device realizes the statistical processing of the user query information of the business system in the method for importing and exporting a large amount of data based on Excel of the present application, constructs a unified query model; creates an asynchronous data processing component, and after encapsulation based on the unified query model, adds it to the business system; the asynchronous data processing component responds to the Excel-based data import or export request, and uses the unified query model to parse the user query information, determines the target operation and data flow direction, and generates a target task; the scheduling service receives the target task to start an asynchronous operation, divides the target task according to the target task and the server resource situation until the target task is completed and feeds back to the front-end page of the business system, obtaining the beneficial effects of improving the data processing efficiency, enhancing the stability of the business system, reducing the integration cost, ensuring data accuracy, and improving the user experience, thereby comprehensively optimizing the process of importing and exporting a large amount of data.

[0043] In the storage medium provided by the present application, there is a program product that can implement the method for importing and exporting a large amount of data based on Excel.

[0044] The method for importing and exporting a large amount of data based on Excel includes: Statistically process the user query information of the business system and construct a unified query model; Create an asynchronous data processing component, and after encapsulation based on the unified query model, add it to the business system; The asynchronous data processing component responds to the Excel-based data import or export request, and uses the unified query model to parse the user query information, determines the target operation and data flow direction, and generates a target task; The scheduling service receives the target task to start an asynchronous operation, divides the target task according to the target task and the server resource situation until the target task is completed and feeds back to the front-end page of the business system.

[0045] In some possible embodiments, the Excel-based large data volume import / export processing method of the present disclosure may be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Method" section above in this specification.

[0046] The storage medium of the present disclosure may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0047] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large data volume import and export processing method based on Excel, characterized in that: The steps include: S1. Collect statistics on user query information of the business system and build a unified query model; S2. Create an asynchronous data processing component, encapsulate it based on the unified query model, and add it to the business system; S3. The asynchronous data processing component responds to Excel-based data import or export requests, and uses a unified query model to parse user query information, determine target operations and data flows, and generate target tasks; S4. The scheduling service receives the target task and starts an asynchronous operation, divides the target task according to the target task and server resource conditions, and feedback is sent to the front-end page of the business system until the target task is completed.

2. The Excel-based large data volume import and export processing method according to claim 1, characterized in that: The specific steps of step S1 are as follows: S11. Collect user query information in the business system; the user query information includes query resources, search conditions and data permissions; S12. Analyze the collected user query information, build a unified data structure and unified query logic, and obtain a unified query model.

3. The Excel-based large data volume import and export processing method according to claim 2, characterized in that: The specific steps of step S12 are as follows: S121. Identify the data types of business objects in different scenarios based on the collected user query information and establish a unified basic data object; the unified basic data object includes general fields and specific business fields that characterize business scenario attributes; S122. Define a unified search condition data structure to describe the user's query requirements; S123. Establish data permissions data structure to store user access rights to different business objects; S124. Unify the basic data objects, the search condition data structure and the data authority data structure as a unified data structure; S125. Define the unified query logic as follows: convert the search conditions in the user query information according to the unified search condition data structure, and verify them using the data authority data structure, then use the converted and verified search conditions to execute a query on the unified basic data object to the data source, and finally process the query results and return them.

4. The Excel-based large data import and export processing method according to claim 3, characterized in that: The specific steps of step S2 are as follows: S21. Select open tools and frameworks based on the programming language and operating environment used by the business system; S22. Use the selected development tools and design asynchronous data processing components based on the selected framework, add data import modules and data export modules, and use tool classes to design data verification modules and logging modules respectively; S23. Bind and encapsulate the asynchronous data processing component with the unified query model, design the interface with the business system, and define the request parameter format as input and the return result format as output; S24. Add the asynchronous data processing component to the business system through the designed interface.

5. The Excel-based large data import and export processing method according to claim 4, characterized in that: The specific steps of step S3 are as follows: S31. After the business system receives the Excel-based data import or export request, it inputs it into the asynchronous data processing component according to the request parameter format; S32. The asynchronous data processing component obtains user query information from the data import or export request, and parses and filters the user query information using a unified query model; S33. The asynchronous data processing component uses the data import module to process the user query information of the data import request, and uses the data export module to process the user query information of the data export request; S34. The asynchronous data processing component verifies the processing result of the user query information using the data verification module, and determines whether the target operation is data import or data export based on the processing result of the user query information that passes the verification; If the target operation is data import, determine the data flow direction to export data from the Excel spreadsheet to the business system; If the target operation is data export, determine the data flow direction to export data from the business system to Excel spreadsheet; S35. Determine that a target operation needs to be performed on the processing result of the user query information, obtain a target task, and send it to the scheduling service.

6. The Excel-based large data import and export processing method according to claim 5, characterized in that: The specific steps of step S32 are as follows: S321. The asynchronous data processing component executes unified query logic; S322. The asynchronous data processing component obtains the user query information, extracts the search conditions, and converts them according to the unified search condition data structure; S323. The asynchronous data processing component uses the data authority data structure to filter the search conditions; S324. The asynchronous data processing component applies the filtered search conditions to the query of the unified basic data object to determine the data that meets the conditions.

7. The Excel-based large data import and export processing method according to claim 6, characterized in that: The specific steps of step S4 are as follows: S41. The scheduling service receives the target task and stores it in the warehouse, starting the asynchronous operation of the business system; S42. The scheduling server starts the target task scheduling; S43. The scheduling service obtains the server resource occupancy rate and determines whether the server resources are idle; If yes, go to step S44; If not, wait for the set time period and return to step S42; S44. The scheduling server obtains the data volume of the target task and determines whether it exceeds the resource load of the server; If yes, return to step S42; If not, proceed to step S45; S45. The scheduling server calculates the number of parallel tasks according to the server resources and the data volume of the target task, and splits the target task into several subtasks according to the number of parallel tasks; S46. The scheduling server allocates server resources to execute the subtasks and merges the execution results of each subtask; S47. The dispatch server generates a message notification in response to the merged data and provides it to the business system front-end page, and converts the merged data into a return result format and provides it to the asynchronous data processing component; S48. The asynchronous data processing component returns the merged data to the business system front-end interface; S49. The front-end interface of the business system responds to the user's data import or export query, performs operations on the merged data, and provides data processing status query and data file download interface.

8. A large data volume import and export processing system based on Excel, characterized in that: include: A unified query model building module is used to collect statistics on user query information of the business system and build a unified query model; Asynchronous data processing component creation module, used to create asynchronous data processing components, and encapsulate them based on the unified query model and add them to the business system; The target task generation module is used to respond to Excel-based data import or export requests through asynchronous data processing components, and use a unified query model to parse user query information, determine target operations and data flows, and generate target tasks; The target task scheduling module is used to start asynchronous operations when the scheduling service receives the target task, divide the target task according to the target task and server resource conditions, and feedback to the front-end page of the business system until the target task is completed.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method implements the steps of the Excel-based large data volume import and export processing method as claimed in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large data volume import and export processing method based on Excel as claimed in any one of claims 1 to 7 are implemented.

Images