Data exporting method, system and equipment and medium

Abstract

The invention discloses a data exporting method, system and device and a medium, applied to a data exporting system, and the method comprises the following steps: receiving an exporting request forwarded by a public basic platform gateway through an exporting application; storing an export task corresponding to the export request, and asynchronously sending an export task message to distributed message middleware; and determining an export batch corresponding to the export task message, and carrying out asynchronous processing on the export task message through the distributed message middleware, the export batch and the export application to generate an export result file. The existing service system can be accessed to the export center by modifying the configuration, the rapid access of the export request can be realized through asynchronous distribution and aggregation of the request message, the interaction time between different processes is reduced, the original query logic is not changed, and meanwhile, the system resource preemption risk when a large amount of data is exported can be avoided.

Description

[0001] This application claims priority based on the invention patent application titled "A Data Export Method, Device and Medium" with the application number 202311870449.9 filed with the China National Intellectual Property Administration on December 29, 2023. The entire content of the above-mentioned Chinese patent application is incorporated herein by reference. Technical Field

[0002] This application relates to the field of data export, and specifically relates to a data export method, system, device and medium. Background Art

[0003] Data export is the process of extracting raw data from its existing format and converting it into a format required by another application. Exporting data is also a way to backup data or move data between two different versions of a program.

[0004] In recent years, with the continuous development of business, the demand for data export by financial and operation personnel has gradually increased. To maintain a good export experience, systems in different fields need to perform frequent and repetitive development for specific export operations. This development process greatly wastes human and physical resources, and it is impossible to control the risk of current system resource occupation when a large amount of data is exported in multiple batches. Summary of the Invention

[0005] To solve the above problems, this application proposes a data export method, system, device and medium. The method is applied to a data export system and includes:

[0006] Receiving an export request forwarded by a common basic platform gateway through an export application; saving the export task corresponding to the export request, and asynchronously sending an export task message to a distributed message middleware; determining the export batch corresponding to the export task message, and asynchronously processing the export task message through the distributed message middleware, the export batch and the export application to generate an export result file.

[0007] Through asynchronous distribution and aggregation of request messages, fast access to export requests can be achieved, reducing the interaction time between different processes, without changing the original query logic, and at the same time avoiding the risk of system resource occupation when a large amount of data is exported. It is generally applicable to export operations in different business fields and avoids repetitive development.

[0008] In one example, before receiving an export request forwarded by a common basic platform gateway through the export application, the method further includes: obtaining an identity token sent by a user terminal through a user center application, and authenticating the user terminal based on the identity token; after successful authentication, returning the user information corresponding to the user terminal to the common basic platform network management.

[0009] By authenticating the client, determining the data permissions of the client, and only exporting the data corresponding to the user permissions, the risk of data leakage can be avoided.

[0010] In one example, the asynchronous processing of the export task message is performed through the distributed message middleware, the export batch, and the export application, which specifically includes: obtaining a synchronous blocking utility class through the data source configuration of the distributed message middleware; the synchronous blocking utility class is used to access the remote Hypertext Transfer Protocol service; based on the synchronous blocking utility class, requesting the data source and updating the total number of export tasks and export batches; constructing task batch information according to the export task information and saving the task batch information; delivering the task batch information to the thread pool and pushing it to the distributed message middleware.

[0011] By using the message middleware as the storage of the task batch information, it is used to implement functions such as transaction messages, sequential messages, batch messages, timed messages, message backtracking, etc., especially the functions of asynchronous decoupling and traffic peak shaving.

[0012] In one example, the data export system further includes: an operation platform order management application, an order business microservice; the requesting the data source and updating the total number of export tasks and export batches based on the synchronous blocking utility class specifically includes: the operation platform order management application obtaining order list information from the order business microservice; the export application calling the order list interface from the operation platform order management application according to the export conditions; updating the total number of export tasks and export batches based on the order list interface.

[0013] In one example, the data export system further includes: a memory-based database; the updating the total number of export tasks and export batches based on the order list interface specifically includes: the export application updating the export task information through the order paging information; sending the updated export task information to the export microservice so that the export microservice constructs task batch information based on the updated export task information and returns it to the export application; after the export application saves the total number of task batches to the memory-based database, processing the task batch information to update the total number of export batches.

[0014] In one example, the processing the task batch information to update the total number of export batches specifically includes: after the export application processes the task batch information, saving the processed task batch information to the export microservice; saving the task batch information to the distributed message middleware, and processing the export of the task batch message based on the order of the task batch information in the distributed message middleware, and saving the batch record to the export microservice.

[0015] In one example, the data export system further includes: a File Transfer Protocol (FTP) server; after saving the batch record to the export microservice, the method further includes: processing a batch request through the export application, and obtaining order list information from the order business microservice through the order management application of the operation platform; constructing a batch note through the export application and saving the batch note to the export microservice; constructing a batch record file through the export application and saving the batch record file to the FTP server; the export application updating batch information to the export microservice based on the batch record file and communicating batch reduction information to the in-memory database.

[0016] In one example, after communicating the batch reduction information to the in-memory database, the method further includes: determining that the total number of batches of the current export task is zero, then iteratively obtaining the export intermediate files corresponding to the current export task through the FTP server; aggregating the export intermediate files to generate an export result file; saving the export result file to the FTP server, and updating the export task status and information.

[0017] By uniformly generating the export result file, it is possible to uniformly monitor the export process and achieve overall control of export risks.

[0018] This application also provides a data export system, including: a receiving module, configured to receive an export request forwarded by a common basic platform gateway through an export application; a sending module, configured to save the export task corresponding to the export request and asynchronously send an export task message to a distributed message middleware; a processing module, configured to determine an export batch corresponding to the export task message, and asynchronously process the export task message through the distributed message middleware, the export batch, and the export application to generate an export result file.

[0019] This application also provides a data export device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the data export method as described in any of the above examples when executing the program.

[0020] This application also provides a non-volatile computer storage medium storing computer-executable instructions, where the computer-executable instructions implement the data export method as described in any of the above examples when executed by a processor.

[0021] The method proposed by this application can bring the following beneficial effects:

[0022] 1. By asynchronously distributing and aggregating request messages, it is possible to achieve quick access to export requests, reduce the interaction time between different processes, and without changing the original query logic. At the same time, it can avoid the risk of system resource preemption when a large amount of data is exported. It is generally applicable to export operations in different business fields and avoids duplicate development.

[0023] 2. By authenticating the user side to determine the data permissions of the user side and only exporting the data corresponding to the user permissions, the risk of data leakage can be avoided.

[0024] 3. By using a message middleware as the storage of task batch information to implement functions such as transaction messages, sequential messages, batch messages, timed messages, and message backtracking, especially the functions of asynchronous decoupling and traffic peak shaving.

[0025] 4. By uniformly generating the export result file, it is possible to uniformly monitor the export process and achieve overall control of the export risk. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0027] Figure 1 is a flowchart of a data export method in an embodiment of the present application;

[0028] Figure 2 is a timing diagram of a data export method in an embodiment of the present application;

[0029] Figure 3 is a timing diagram of processing an export task message in an embodiment of the present application;

[0030] Figure 4 is a timing diagram of processing an export task batch message in an embodiment of the present application;

[0031] Figure 5 is a timing diagram of data iterative export in an embodiment of the present application;

[0032] Figure 6 is a structural diagram of a data export device in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following will clearly and completely describe the technical solutions of this application in combination with specific embodiments of this application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0034] The following will, in combination with the drawings, detail the technical solutions provided by each embodiment of this application.

[0035] Figure 1 It is a schematic flowchart of a data export method provided for one or more embodiments of this specification. This method can be applied to different business fields, such as the Internet finance business field, the e-commerce business field, the instant messaging business field, the game business field, the official business field, etc. This process can be executed by computing devices in the corresponding fields (such as the risk control server or intelligent mobile terminal corresponding to the payment business, etc.), and some input parameters or intermediate results in the process allow manual intervention and adjustment to help improve accuracy.

[0036] The implementation of the analysis method involved in the embodiments of this application can be a terminal device or a server, and this application does not make special restrictions on this. For the convenience of understanding and description, the following embodiments will be described in detail using a server as an example.

[0037] It should be noted that this server can be a single device or a system composed of multiple devices, that is, a distributed server, and this application does not make specific limitations on this.

[0038] As Figures 1 - 5 shown, the embodiments of this application provide a data export method applied in a data export system. The data export system includes: a user side, a common basic platform gateway, an export application, an export microservice, and a distributed message middleware; the method includes:

[0039] S101: Receive, through the export application, an export request forwarded by the common basic platform gateway.

[0040] In one embodiment, before the export application receives the export request, it is also necessary to authenticate the user center application. Specifically, through the user center application, obtain the identity token sent by the user side, and authenticate the user side based on the identity token; after the authentication passes, return the user information corresponding to the user side to the common basic platform network management. By authenticating the user side, determine the data permissions of the user side, and only export the data corresponding to the user permissions, which can avoid the risk of data leakage.

[0041] By authenticating the client, determining the data permissions of the client, and only exporting the data corresponding to the user permissions, the risk of data leakage can be avoided.

[0042] S102: Save the export task corresponding to the export request and asynchronously send the export task message to the distributed message middleware.

[0043] S103: Determine the export batch corresponding to the export task message, and asynchronously process the export task message through the distributed message middleware, export batch, and export application to generate an export result file.

[0044] In one embodiment, when performing asynchronous processing, the distributed message middleware obtains a synchronous blocking utility class according to the data source configuration; the synchronous blocking utility class here is used to access the remote Hypertext Transfer Protocol service; then based on the synchronous blocking utility class, request the data source, and update the total number of export tasks and export batches; construct task batch information according to the export task information, and save the task batch information; deliver the task batch information to the thread pool and push it to the distributed message middleware.

[0045] By using the message middleware as the storage of task batch information, it is used to implement functions such as transaction messages, sequential messages, batch messages, timed messages, message backtracking, etc., especially the functions of asynchronous decoupling and traffic peak shaving.

[0046] Furthermore, the data export system further includes: an operation platform order management application, an order business microservice. When requesting the data source based on the synchronous blocking utility class and updating the total number of export tasks and export batches, the operation platform order management application obtains order list information from the order business microservice; the export application calls the order list interface from the operation platform order management application according to the export conditions; based on the order list interface, update the total number of export tasks and export batches.

[0047] Furthermore, the data export system further includes: a memory database; when updating the total number of export tasks and export batches based on the order list interface, the export application updates the export task information through the order paging information; then sends the updated export task information to the export microservice, so that the export microservice constructs task batch information based on the updated export task information and returns it to the export application; after the export application saves the total number of task batches to the memory database, processes the task batch information to update the total number of export batches.

[0048] Further, when updating the total number of export batches, after the export application processes the task batch information, the processed task batch information is saved to the export microservice, and the task batch information is saved to the distributed message middleware. Based on the order of the task batch information in the distributed message middleware, the export task batch messages are processed, and the batch records are saved to the export microservice.

[0049] In one embodiment, the data export system further includes: a File Transfer Protocol (FTP) server; after saving the batch records to the export microservice, the export application processes the batch request, and through the operation platform order management application, obtains the order list information, such as the battery swapping order information of the battery swapping station, from the order business microservice. Then, the export application constructs a batch note and saves the batch note to the export microservice. Next, the export application constructs a batch record file and saves the batch record file to the FTP server. Among them, based on the batch record file, the export application updates the batch information to the export microservice and conveys the batch reduction information to the in-memory database.

[0050] In one embodiment, when the total number of batches is zero, the current export task is completed, and an export result file needs to be generated. When generating, it is determined that the total number of batches of the current export task is zero, then the export intermediate files corresponding to the current export task are iteratively obtained through the FTP server; the export intermediate files are aggregated to generate an export result file; the export result file is saved to the FTP server, and the export task status and information are updated.

[0051] By uniformly generating the export result file, the export process can be uniformly monitored, and overall control of the export risk is achieved.

[0052] Through the asynchronous distribution and aggregation of request messages, this application can achieve the fast access of export requests, reduce the interaction time between different processes, and does not change the original query logic. At the same time, it can avoid the risk of system resource preemption when a large amount of data is exported. It is generally applicable to export services in different business fields and avoids repeated development.

[0053] An embodiment of this application further provides a data export system, including:

[0054] A receiving module, configured to receive, through the export application, an export request forwarded by the common basic platform gateway.

[0055] A sending module, configured to save the export task corresponding to the export request and asynchronously send the export task message to the distributed message middleware.

[0056] A processing module, configured to determine the export batch corresponding to the export task message and asynchronously process the export task message through the distributed message middleware, the export batch, and the export application to generate an export result file.

[0057] As Figure 6 shown, an embodiment of the present application further provides a data export device, including:

[0058] At least one processor; and,

[0059] A memory communicatively connected to the at least one processor; wherein,

[0060] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data export method described in any of the above embodiments.

[0061] In an embodiment of the present application, the above processor can execute:

[0062] Receive an export request forwarded by a common basic platform gateway through an export application; save the export task corresponding to the export request, and asynchronously send an export task message to a distributed message middleware; determine the export batch corresponding to the export task message, and asynchronously process the export task message through the distributed message middleware, the export batch, and the export application to generate an export result file.

[0063] An embodiment of the present application further provides a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are configured to implement the data export method described in any of the above embodiments.

[0064] In an embodiment of the present application, the above computer-executable instructions are configured to be able to implement:

[0065] Receive an export request forwarded by a common basic platform gateway through an export application; save the export task corresponding to the export request, and asynchronously send an export task message to a distributed message middleware; determine the export batch corresponding to the export task message, and asynchronously process the export task message through the distributed message middleware, the export batch, and the export application to generate an export result file.

[0066] The embodiments in the present application are all described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and the relevant parts can refer to the partial descriptions of the method embodiments.

[0067] The devices, media, and methods provided by the embodiments of the present application correspond one by one. Therefore, the devices and media also have beneficial technical effects similar to those of their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be elaborated here.

[0068] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more flows and / or Figure 1 blocks or multiple blocks.

[0070] These computer program instructions can also be stored in a computer-readable memory capable of guiding the computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one or more flows and / or Figure 1 blocks or multiple blocks.

[0071] These computer program instructions can also be loaded onto the computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one or more flows and / or Figure 1 blocks or multiple blocks.

[0072] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0073] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.

[0074] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transitory media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0075] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0076] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A data export method, characterized in that, Applied to a data export system, the method includes: Receiving an export request forwarded by a common basic platform gateway through an export application; Saving the export task corresponding to the export request and asynchronously sending an export task message to a distributed message middleware; Determining an export batch corresponding to the export task message, and asynchronously processing the export task message through the distributed message middleware, the export batch, and the export application to generate an export result file.

2. The method according to claim 1, characterized in that, Before receiving the export request forwarded by the common basic platform gateway through the export application, the method further includes: Obtaining an identity token sent by a user terminal through a user center application and authenticating the user terminal based on the identity token; After successful authentication, returning the user information corresponding to the user terminal to the common basic platform network management.

3. The method according to claim 1, characterized in that, The asynchronous processing of the export task message through the distributed message middleware, the export batch, and the export application specifically includes: Obtaining a synchronous blocking utility class through the data source configuration of the distributed message middleware; the synchronous blocking utility class is used to access a remote hypertext transfer protocol service; Requesting a data source based on the synchronous blocking utility class and updating the total number of export tasks and export batches; Constructing task batch information according to export task information and saving the task batch information; Delivering the task batch information to a thread pool and pushing it to the distributed message middleware.

4. The method according to claim 3, wherein The data export system further includes: an operation platform order management application, an order service microservice; The requesting a data source based on the synchronous blocking utility class and updating the total number of export tasks and export batches specifically includes: The operation platform order management application obtaining order list information from the order service microservice; The export application calling an order list interface from the operation platform order management application according to export conditions; Updating the total number of export tasks and export batches based on the order list interface.

5. The method according to claim 4, wherein The data export system further includes: a memory database; The updating the total number of export tasks and export batches based on the order list interface specifically includes: The export application updating export task information through the order paging information; Sending the updated export task information to the export microservice so that the export microservice constructs task batch information based on the updated export task information and returns it to the export application; After the export application saves the total number of task batches to the memory database, processing the task batch information to update the total number of export batches.

6. The method according to claim 5, characterized in that The processing the task batch information to update the total number of export batches specifically includes: After processing the task batch information through the export application, saving the processed task batch information to the export microservice; Saving the task batch information to the distributed message middleware, and processing the export of the task batch message based on the order of the task batch information in the distributed message middleware and saving the batch record to the export microservice.

7. The method according to claim 6, characterized in that, The data export system further includes: a file transfer protocol server; After saving the batch record to the export microservice, the method further includes: Processing a batch request through the export application, and obtaining order list information from the order business microservice through the order management application of the operation platform; Constructing a batch note through the export application, and saving the batch note to the export microservice; Constructing a batch record file through the export application, and saving the batch record file to the File Transfer Protocol (FTP) server; Based on the batch record file, the export application updates batch information to the export microservice and conveys batch reduction information to the in-memory database.

8. The method according to claim 7, wherein After conveying the batch reduction information to the in-memory database, the method further includes: Determining that the total number of batches of the current export task is zero, then iteratively obtaining the export intermediate file corresponding to the current export task through the FTP server; Aggregating the export intermediate files to generate an export result file; Saving the export result file to the FTP server, and updating the export task status and information.

9. A data export system, characterized in that, including: A receiving module, configured to receive an export request forwarded by a common basic platform gateway through an export application; A sending module, configured to save the export task corresponding to the export request and asynchronously send an export task message to a distributed message middleware; A processing module, configured to determine an export batch corresponding to the export task message, and asynchronously process the export task message through the distributed message middleware, the export batch, and the export application to generate an export result file.

10. A data export device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the data export method according to any one of claims 1-8 when executing the program.

11. A non - volatile computer storage medium stores computer - executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, the data export method according to any one of claims 1-8 is implemented.

Images