File export system and method, computer readable storage medium and electronic equipment
By introducing the communication mechanism between the business service client and the file export server in the file export system, and generating and executing file export tasks, the problem of excessive burden on the file export server in the microservice deployment scenario is solved, and the system stability and efficiency are improved.
Patent Information
- Application Number
- CN202411524049.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-02-18
AI Technical Summary
In the business scenario of microservice deployment, the file export server is overloaded.
By introducing the communication mechanism between the business service client and the file export server in the file export system, a file export request is generated, and a file export task is executed based on the preset task execution thread, avoiding full export at once.
It reduces the burden on the file export server, improves the stability of the file export system, and simplifies the mutual calls between services and improves file export efficiency by decoupling business services and file export services.
Smart Images
Figure CN119484505A_ABST
Abstract
Description
Background Art
[0002] In various business scenarios, especially financial and e-commerce scenarios, it is often necessary to perform big data analysis based on the business data generated in the scenario, so large files need to be exported frequently.
[0003] However, since most of the above business scenarios are deployed in a microservices manner, that is, different business scenarios correspond to different servers; under this premise, exporting all business files in a server will place a heavy burden on the file export server.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] The purpose of the present disclosure is to provide a file export system, a file export method, a computer-readable storage medium and an electronic device, thereby at least to a certain extent overcoming the problem of heavy burden on the file export server caused by the limitations and defects of the relevant technology.
[0006] According to one aspect of the present disclosure, there is provided a file export system, comprising:
[0007] A business service client and a file export server, wherein the business service client is in communication connection with the file export server, wherein:
[0008] The business service client is used to determine the file parameters to be exported, and generate a file export request according to the file parameters, the business service interface to be called to export the file parameters, and the interface call type of the business service interface;
[0009] The file export server is used to receive a file export request sent by the business service client, and determine whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request; and
[0010] When it is determined that the business service interface can be accessed normally, one or more file export tasks corresponding to the business service client are created according to the data size and the file uniform resource locator, and one or more file export tasks are executed based on a preset task execution thread to obtain a business file corresponding to the business service client.
[0011] In an exemplary embodiment of the present disclosure, the file export server is a server with a large file export function, and the business service client is a micro server corresponding to the application.
[0012] In an exemplary embodiment of the present disclosure, the business service client is further used for:
[0013] Importing preset parameter annotations into the original business file in the business service client; wherein the parameter annotations include at least one of a time format parameter annotation, an enumeration value format parameter annotation, a table style parameter annotation, and a field parameter annotation to be exported;
[0014] All class attribute information of all classes included in the original business file is obtained, and all class attribute information of all classes is parsed based on the imported parameter annotations to obtain file parameters that need to be exported.
[0015] In an exemplary embodiment of the present disclosure, the business service client is further used for:
[0016] Acquire the time attributes, enumeration value format attributes, table style attributes, and field attributes to be exported included in the original business file in the business service client;
[0017] Importing the time format parameter annotation into the time class attribute, and importing the enumeration value format parameter annotation into the enumeration value format class attribute;
[0018] Injecting the table style parameter annotation into the table style class attribute; wherein the table style parameter annotation includes at least one of a table width annotation, a table header height annotation, and a table content height annotation;
[0019] Inject the field parameter annotation that needs to be exported into the field class attribute that needs to be exported; wherein the field parameter annotation includes the table header alias of the data table that needs to be exported and / or the export order of the data table.
[0020] In an exemplary embodiment of the present disclosure, the business service client is further used for:
[0021] Parsing the timestamp data corresponding to the time attribute based on the time format parameter annotation to obtain the time parameter, and parsing the enumeration data corresponding to the enumeration value format attribute based on the enumeration value format parameter annotation to obtain the enumeration data parameter;
[0022] Parsing the table style class attributes based on the table style parameter annotation to obtain the table class parameters, and parsing the data corresponding to the field class attributes to be exported based on the field parameter annotation to be exported to obtain the field parameters to be exported;
[0023] The file parameters to be exported are generated according to the time parameters, enumeration data parameters, table parameters and field parameters to be exported.
[0024] In an exemplary embodiment of the present disclosure, the file export server is further used for:
[0025] Parsing the file export request to obtain the file request method and the file uniform resource locator of the business service client, and calling the business service interface of the business service client according to the file request method and the file uniform resource locator;
[0026] According to the interface return information fed back by the business service interface, it is determined whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client.
[0027] In an exemplary embodiment of the present disclosure, the file export server is further used for:
[0028] Determining the file parameters to be exported according to the file uniform resource locator, and determining the sub-data volume of each sub-parameter in the file parameters to be exported according to the data volume;
[0029] Generate one or more file export tasks according to the parameter name of the sub-parameter, the sub-data volume, and the interface call type of the business service interface;
[0030] One or more file export tasks are placed in a task cache queue, and each of the file export tasks in the task cache queue is executed based on a preset task execution thread to obtain a business file corresponding to the business service client.
[0031] In an exemplary embodiment of the present disclosure, the interface call type includes a hypertext transfer protocol type and / or a remote procedure call type; wherein the file export server is further used to:
[0032] When the interface call type is a remote procedure call type, determining a Feign client corresponding to the business service client according to the business service name of the business service client;
[0033] Based on the Feign client, the business file is obtained from the business service interface according to the data request mode of the business service interface.
[0034] In an exemplary embodiment of the present disclosure, the file export server is further used for:
[0035] According to the file uniform resource locator of the business service client, the data request mode of the business service interface is determined; wherein the data request mode includes a Get request mode and / or a Post request mode.
[0036] According to one aspect of the present disclosure, a file export method is provided, which is configured on a file export server, and the file export method includes:
[0037] Receive a file export request sent by a business service client, and determine whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request;
[0038] When it is determined that the business service interface can be accessed normally, one or more file export tasks corresponding to the business service client are created according to the data volume and the file uniform resource locator;
[0039] One or more file export tasks are executed based on a preset task execution thread to obtain a business file corresponding to the business service client.
[0040] In an exemplary embodiment of the present disclosure, one or more file export tasks corresponding to the business service client are created according to the data size and the file uniform resource locator, including:
[0041] Determining the file parameters to be exported according to the file uniform resource locator, and determining the sub-data volume of each sub-parameter in the file parameters to be exported according to the data volume;
[0042] Generate one or more file export tasks according to the parameter name of the sub-parameter, the sub-data volume, and the interface call type of the business service interface;
[0043] One or more file export tasks are placed in a task cache queue, and each of the file export tasks in the task cache queue is executed based on a preset task execution thread to obtain a business file corresponding to the business service client.
[0044] In an exemplary embodiment of the present disclosure, determining whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request includes:
[0045] Parsing the file export request to obtain the file request method and the file uniform resource locator of the business service client, and calling the business service interface of the business service client according to the file request method and the file uniform resource locator;
[0046] According to the interface return information fed back by the business service interface, it is determined whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client.
[0047] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the file export method described in any one of the above is implemented.
[0048] According to one aspect of the present disclosure, there is provided an electronic device, including:
[0049] Processor; and
[0050] A memory, configured to store executable instructions of the processor;
[0051] The processor is configured to execute any one of the above-mentioned file export methods by executing the executable instructions.
[0052] The file export system provided by the embodiment of the present disclosure, on the one hand, can determine the file parameters to be exported through the business service client, and generate a file export request according to the file parameters, the business service interface to be called for exporting the file parameters, and the interface call type of the business service interface; then the file export server is used to receive the file export request sent by the business service client, and determine whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request; at the same time, when it is determined that the business service interface can be normally accessed, one or more file export tasks corresponding to the business service client are created according to the data volume and the file uniform resource locator, and one or more file export tasks are executed based on a preset task execution thread to obtain the business file corresponding to the business service client. Since there is no need to export all the business files at one time, the problem in the related art that the file export server is heavily burdened due to the need to export all the business files in a server at one time is solved, the burden of the file export server is reduced, and the stability of the file export system is improved; on the other hand, since the business service can be used as a client, the decoupling between the business service and the file export service is achieved, the complexity of calling each other between the services is reduced, and the file export efficiency is improved.
[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0055] Figure 1 The following schematically shows an example architecture diagram of a file export system according to an example embodiment of the present disclosure.
[0056] Figure 2 An example diagram of a multi-business scenario of a file export system according to an example embodiment of the present disclosure is schematically shown.
[0057] Figure 3 A flowchart of a method for implementing parameter annotation of a business service client according to an exemplary embodiment of the present disclosure is schematically shown.
[0058] Figure 4 A scenario example diagram schematically illustrates an import result of @ExportField according to an exemplary embodiment of the present disclosure.
[0059] Figure 5 The flowchart schematically shows a process of creating file parameters of a business service client according to an exemplary embodiment of the present disclosure.
[0060] Figure 6 A diagram schematically illustrates an example scenario of a file export task established according to an example embodiment of the present disclosure.
[0061] Figure 7 A scenario example diagram schematically illustrates an established task queue and a specific execution process of a task according to an example embodiment of the present disclosure.
[0062] Figure 8 The following is a flowchart schematically showing a method for exporting a file according to an exemplary embodiment of the present disclosure.
[0063] Fig. 9 A block diagram schematically illustrates a file exporting device according to an exemplary embodiment of the present disclosure.
[0064] Fig.10 An electronic device for implementing a file exporting method according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0065] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.
[0066] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0067] The exemplary embodiment of the present disclosure first provides a file export system. Specifically, refer to Figure 1 As shown, the file export system may include a business service client 110 and a file export server 120, and the business service client and the file export server may be connected to each other via a wired network or a wireless network. In actual application, the business service client may be used to determine the file parameters to be exported, and generate a file export request according to the file parameters, the business service interface to be called to export the file parameters, and the interface call type of the business service interface. The file export server may be used to receive the file export request sent by the business service client, and determine whether the business service interface can be accessed normally and the data volume of the business file to be exported in the business service client according to the file export request. Furthermore, when it is determined that the business service interface can be accessed normally, one or more file export tasks corresponding to the business service client are created according to the data volume and the file uniform resource locator, and one or more file export tasks are executed based on a preset task execution thread to obtain a business file corresponding to the business service client.
[0068] In the above-mentioned file export system, on the one hand, since the file parameters to be exported can be determined through the business service client, and a file export request is generated according to the file parameters, the business service interface to be called for exporting the file parameters, and the interface call type of the business service interface; then the file export server is used to receive the file export request sent by the business service client, and determine whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request; at the same time, when it is determined that the business service interface can be normally accessed, one or more file export tasks corresponding to the business service client are created according to the data volume and the file uniform resource locator, and one or more file export tasks are executed based on a preset task execution thread to obtain the business file corresponding to the business service client. Since there is no need to export all the business files at one time, the problem in the related art that the file export server is heavily burdened due to the need to export all the business files in a server at one time is solved, the burden of the file export server is reduced, and the stability of the file export system is improved; on the other hand, since the business service can be used as a client, the decoupling between the business service and the file export service is achieved, the complexity of mutual calls between services is reduced, and the file export efficiency is improved.
[0069] Hereinafter, the file export system described in the exemplary embodiment of the present disclosure will be explained and illustrated in detail with reference to the accompanying drawings.
[0070] First, the technical implementation principle of the example embodiment of the present disclosure is explained and illustrated. Specifically, in various business systems, especially in scenarios such as finance, e-commerce, and big data analysis that require frequent large file exports, most current services are deployed in the form of microservices, and different business services correspond to different servers. The file export of multiple scenarios will not only bring redundant code, but also the large size of the exported file will occupy a large amount of CPU (Central Processing Unit) resources, and in severe cases, it will even cause the entire server to crash. Under this premise, in order to reduce the redundancy of the code and improve the security and stability of the entire system, the example embodiment of the present disclosure proposes a file export system. In this file export system, the large file export function is separately encapsulated as an integral file export server, and each business server is used as a business service client corresponding to the file export server, and the file export service is uniformly asynchronously called to implement the file export function, thereby achieving the improvement of the security and stability of the entire file export system on the basis of reducing the redundancy of the code.
[0071] Furthermore, since the file export server needs to obtain and parse data from the original business server, the data definition and parameter definition of the file to be exported need to be specified in this process; therefore, in order to simplify the data definition process of the business service client when it is called, the file export system also provides a solution for defining exported data; specifically, in the actual application process, since the file export server provides the analysis of defining export data and querying data parameters, the business service client only needs to simply use annotations on the attributes to complete the data definition.
[0072] Furthermore, for the transmission methods of HTTP (Hypertext Transfer Protocol) or RPC (Remote Procedure Call) commonly used in the calling process between the business service client and the file export server, the file export system also implements a strategy for dynamically generating FeignClient for Feign using RPC as the calling method; that is, in the actual application process, the file export server does not need to explicitly define the FeignClient of different business services in its own code, but only needs to use lazy loading to dynamically create it when obtaining data.
[0073] Finally, after the business service client uses the SDK to define attributes, the file export server uses multi-threaded calls to obtain data. It can then parse the data to be exported according to the attribute definition, and then merge the files generated by each thread and upload them to the file storage server to generate a file download link. In addition, by splitting the export of large files into multiple subtasks and generating files asynchronously, the generated files are also sent to the distributed file storage server in a multi-threaded manner, and a file download link is generated and returned to the client, so as to improve the file export efficiency while reducing the burden on the file export server.
[0074] The following will Figure 1 The file export system shown in is further explained and illustrated.
[0075] In an exemplary embodiment, the file export server described above is a server with a large file export function, and the business service client described above is a micro server corresponding to the application program; further, refer to Figure 2As shown, the business service clients recorded above may include one or more, and each application corresponding to a different business scenario has a business service client; for example, the microserver corresponding to the application of the financial scenario corresponds to a business service client; for another example, the microserver corresponding to the application of the e-commerce scenario corresponds to a business service client; for another example, the microserver corresponding to the application of the learning scenario corresponds to a business service client, and so on; in the actual application process, the number of business service clients can be set according to actual needs, and there is no special restriction here.
[0076] In an exemplary embodiment, the business service client can determine the file parameters to be exported by the following method: importing a preset parameter annotation into the original business file in the business service client; wherein the parameter annotation includes at least one of a time format parameter annotation, an enumeration value format parameter annotation, a table style parameter annotation, and a field parameter annotation to be exported; obtaining all class attribute information of all classes included in the original business file, and parsing all class attribute information of all classes based on the imported parameter annotation to obtain the file parameters to be exported. That is, in the actual application process, the file export server can first provide a SDK (Software Development Kit) with an annotation function for the business service client, and then inject the SDK into the business service client in the form of annotations, so that the business service client can quickly and accurately parse the fields to be exported, and format the fields to be exported, so that the business file has a unified format type during the export process.
[0077] In an exemplary embodiment, referring to Figure 3 As shown, the business service client can include the following steps in the process of parameter annotation:
[0078] Step S310, obtaining time attributes, enumeration value format attributes, table style attributes, and field attributes to be exported included in the original business file in the business service client;
[0079] Step S320, importing the time format parameter annotation into the time type attribute, and importing the enumeration value format parameter annotation into the enumeration value format type attribute;
[0080] Step S330, injecting the table style parameter annotation into the table style class attribute; wherein the table style parameter annotation includes at least one of a table width annotation, a table header height annotation, and a table content height annotation;
[0081] Step S340, injecting the field parameter annotation to be exported into the field class attribute to be exported; wherein the field parameter annotation includes the table header alias of the data table to be exported and / or the export order of the data table.
[0082] Steps S310 to S340 will be explained and illustrated below. Specifically, the SDK annotations recorded above may include formatting annotations, EXCEL table style annotations, export property annotations, and ExportParam annotations, etc.; wherein, the formatting annotations recorded here may include time format parameter annotations and enumeration value format parameter annotations; the EXCEL table style annotations recorded here (i.e., table style parameter annotations) may include table width annotations, table header height annotations, and table content height annotations, etc.
[0083] In an example embodiment, in the actual application process, first, the specific injection process of the time format annotation @DateTimeFormat is: enter the time format in the time field; when the time format is entered, it will be parsed according to the format when the text is subsequently exported; for example, the time format is uniformly parsed into YYMMDDHHMMSS (year-month-day-hour-minute-second); secondly, the specific import process of the enumeration value format annotation @ValueEnumFormat is: enter the enumeration value format in the field whose value is the enumeration type value, and the corresponding enumeration value will be found according to the attribute value when the text is subsequently exported; then, the table width annotation @ExcelColumnWidth can specify the subsequent export of EXCEL Table width; further, the header height annotation @ExcelHeadRowHeight can specify the subsequent header height, and the table content height annotation can specify the subsequent table content height; based on this, when exporting files, the corresponding table files can be exported based on the specified header height, table content height, and table width; further, for the export attribute annotation @ExportField, first, this attribute annotation can be injected into the field to be exported to specify a certain field as the field to be exported. There are two attributes in this annotation, title represents the header alias of the exported data row, and order represents the column order after export; among them, the import result of @ExportField can be referred to Figure 4 As shown; when exporting a file, the table to be imported into the data row can be determined based on the header alias of the data row, and the file can be exported based on the corresponding column order; finally, for @ExportParam, it can be used to parse the parameters when requesting an API (Application Programming Interface) (that is, the file parameters that need to be exported as recorded in the previous article).
[0084] In an exemplary embodiment, referring to Figure 5 As shown, the business service client can generate the file parameters to be exported in the following ways:
[0085] Step S510, parsing the timestamp data corresponding to the time attribute based on the time format parameter annotation to obtain the time parameter, and parsing the enumeration data corresponding to the enumeration value format attribute based on the enumeration value format parameter annotation to obtain the enumeration data parameter;
[0086] Step S520, parsing the table style class attributes based on the table style parameter annotation to obtain table class parameters, and parsing the data corresponding to the field class attributes to be exported based on the field parameter annotation to be exported to obtain the field parameters to be exported;
[0087] Step S530, generating file parameters to be exported according to the time parameters, enumeration data parameters, table parameters and field parameters to be exported.
[0088] In the following, steps S510 to S530 are explained and illustrated. Specifically, since only one tool class is provided in the SDK annotation, during the annotation parsing process, only the class to be exported needs to be used as a parameter to parse out all class attributes and attribute types of all classes, and then format and export to obtain the corresponding file parameters. Furthermore, the specific parsing process can be implemented in the following way: first, extract the attributes of all class declarations included in the original business file of the business service client according to Java reflection; secondly, determine in turn whether all attributes contain the @ExportField annotation (that is, determine which fields are the fields that need to be exported); then, determine whether there is @DateTimeFormat on the field attributes that need to be exported, if so, extract the timestamp data in the corresponding format and record it; further, determine whether there is a time format @ValueEnumFormat on the field attributes that need to be exported, if so, parse out all enumeration values and record them; further, according to each field attribute @ExportField that needs to be exported, determine the header alias and export order value corresponding to the field attribute; further, sort the header according to the export order and put it into the parameters of the request file service; finally, parse the file parameters included in the URL corresponding to the file export request of the business service client according to @ExportParam, extract and record them.
[0089] In an example embodiment, after obtaining the file parameters that need to be exported, a file export request can be generated based on the file parameters, the business service interface that needs to be called to export the file parameters, and the interface call type of the business service interface; when the file export request is generated, the file export request can be pushed to the file export server.
[0090] In an example embodiment, after the file export server receives the file export request, it can determine whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client according to the file export request; the specific implementation process can be achieved in the following way: first, the file export request is parsed to obtain the file request method and file uniform resource locator of the business service client, and the business service interface of the business service client is called according to the file request method and file uniform resource locator; secondly, according to the interface return information fed back by the business service interface, it is determined whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client. Specifically, in the actual application process, when the file export server receives the file export service request sent by the business service client, it will first create a file export task. After the creation is completed, a special thread will be executed in the background to execute the file export task to ensure the stability of the service; and in order to prevent the risk of downtime of the file export server caused by directly requesting the full amount of data due to the excessive amount of data of the business file in the business service end, the file export server limits the business service interface to only use the paging interface; under this premise, when the file export server receives the file export service request sent by the business service client, it will first call the paging interface (that is, the business service interface) according to the mode and uniform resource locator of the file export service request. This interface call is only to test whether the business service interface can be accessed and determine the amount of data of the business file to be exported, so the first paging size can be set smaller; further, after waiting for the interface request to obtain a return value (that is, the interface return information), it can be determined based on the return value whether the business service interface can be normally accessed, as well as the amount of data and the total amount of data of the attribute values corresponding to each parameter.
[0091] In an example embodiment, if the business service interface can be accessed normally, a corresponding file export task needs to be created; specifically, it can be implemented in the following manner: first, the file parameters to be exported are determined according to the file uniform resource locator, and the sub-data volume of each sub-parameter in the file parameters to be exported is determined according to the data volume; secondly, one or more file export tasks are generated according to the parameter name of the sub-parameter, the sub-data volume and the interface call type of the business service interface; then, one or more file export tasks are placed in a task cache queue, and each of the file export tasks in the task cache queue is executed based on a preset task execution thread to obtain a business file corresponding to the business service client. That is, in the process of file export task, it is necessary to take into account the sub-data volume of each sub-parameter, and then determine the CPU resources required in the task export process according to the sub-data volume of the sub-parameter, and then divide the file export task into one or more file export tasks according to the required CPU resources; at the same time, each subtask may include data corresponding to one or more sub-parameters, which can be determined according to the CPU resources required for each parameter, and this example does not impose special restrictions on this; further, when the file export task is established, the file execution task can be placed in the corresponding task queue, and the task will be executed by a unified task execution thread; wherein, the established file export task can refer to Figure 6 As shown, the corresponding task queue and the specific execution process of the task can be referred to Figure 7 shown.
[0092] In an exemplary embodiment, since the interface call type recorded above can include the Hypertext Transfer Protocol HTTP type and the Remote Procedure Call RPC type; therefore, in the specific task execution process, it can also be implemented based on different methods; wherein, when the interface call type is the HTTP type, the corresponding business file can be directly obtained from the database corresponding to the business service client through the business service interface based on the HTTP protocol; when the interface call type is the RPC type, the business file can be exported in the following manner: first, according to the business service name of the business service client, the Feign client corresponding to the business service client is determined; secondly, based on the Feign client, according to the data request method of the business service interface, the business file is obtained from the business service interface. Wherein, the data request method is obtained in the following manner: according to the file uniform resource locator of the business service client, the data request method of the business service interface is determined; wherein, the data request method includes a Get request method and / or a Post request method.
[0093] The following will further explain and illustrate the specific execution process of the file export task. Specifically, in the actual application process, the execution of the file export task can use a multi-threaded method to call the business service and parse it to improve the efficiency of obtaining data; at the same time, in order to ensure the order of writing files, the parsed data will be pushed to a unified write file queue. When the write file thread is idle, it will find the data for the next execution in order according to the current completion progress; further, the overall execution process is as follows: ① Obtain the task lock of each file export task, and lock the thread executing the file export task and the file export task based on the task lock, and automatically unlock it after the task is completed, so as to prevent multiple threads from executing repeatedly and causing task execution confusion; ② Obtain the task details information of each file export task from the task cache queue. If not, take out the task details information of the file export task from the task storage database and place it in the task cache queue; ③ Change the task status of the file export task in the task cache queue to RUNNING, and update the task status to the task storage database synchronously; ④ According to the number of file export tasks of the business service client, assign one or more thread workers to the file export task 1. Call the business to obtain the business file corresponding to the file export task; 2. According to the information in the file export task, obtain the request method of the business file. If the request method is RPC, obtain the corresponding FeignClient from the memory according to the business service name. If it is not found in the memory, dynamically create a FeignClient according to the service name and store it in the memory, and then obtain the business file based on FeignClient; if it is HTTP, directly obtain the business file; 3. Call the business service interface of the business service client to obtain the export data; 4. Parse the obtained business files and write them to their respective tables (determine the table to be written and write it according to the header alias of the business file); 5. The file queue writes the tasks to the file in turn, and uses EasyPoi technology to write each data as a line in turn into the file line by line; 6. Update the status of the table after writing the business file, and judge the checkpoint of the business file written in the table; 7. Determine whether the business file is obtained based on the checkpoint. If not, loop step 4; if completed, upload the file to minio and update the status.
[0094] It should be noted that HTTP and RPC are often used between microservices. At the same time, in the relevant enterprise microservice architecture, microservices under the same gateway are often called in Feign. Therefore, when using it, it is necessary to explicitly define the FeignClient of the called party on the caller. The large file export server proposed in the example embodiment of the present disclosure is a general service. It is obviously not appropriate to define the FeignClient of other business services within the service. Therefore, in order to decouple the file export function from each business service, the example embodiment of the present disclosure implements a method of dynamically generating the FeignClient of the called party on the caller. Specifically, in the actual application process, the large file export server When the server calls the business service client to obtain data, it first queries the memory to see whether the FeignClient of the business service client has been generated according to the service name of the called party (business service client) in the gateway; if not, first create a general FeignClient according to the service name of the business service client; then, according to the Restful coding style, the data acquisition interface often uses the Get request method, but considering that the Get request has a limit on the character length of the request URL, there are also many data acquisition interfaces that use Post requests; therefore, the dynamic FeignClient defined in the example embodiment of the present disclosure has only two methods, which are used for Get and Post requests respectively. Different from the methods defined in the traditional FeignClient, the Get and Post request methods defined in the example embodiment of the present disclosure, because it is impossible to predict the specific method and URL called by each service, so the Get and Post methods defined in the FeignClient of the example embodiment of the present disclosure need to pass the calling URL.
[0095] It is necessary to further explain here that, since the business service client has specified the format for data export when calling the large file export server, in the above step ⑦, you only need to write the data line by line according to the format; at the same time, when you first obtain the data, you need to create a file and set the first line of the file as the header. Here, you can use the @ExportField and order in the SDK to define the order and output it in sequence; further, after obtaining the data, you cannot see the type of each data, so you need to determine whether the attribute value can match the attribute enumeration value and time format in turn. If they can match, convert them first and then output them to the table.
[0096] So far, the file export system described in the exemplary embodiment of the present disclosure has been fully implemented. Based on the above content, it can be known that the file export system described in the present disclosure, on the one hand, implements a general file export service and an export data parser, which can define the export field by using simple annotations; on the other hand, it also implements a general solution for large file export, simplifies redundant code, and prevents poor user experience or service lag caused by excessive data.
[0097] This exemplary embodiment also provides a file export method, which can be run on a server, server cluster or cloud server where the file export system is located; of course, those skilled in the art can also run the method disclosed in this disclosure on other platforms as required, and this exemplary embodiment does not specifically limit this. Figure 8 As shown, the file export method may include the following steps:
[0098] Step S810, receiving a file export request sent by the business service client, and determining whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request;
[0099] Step S820, when it is determined that the business service interface can be accessed normally, one or more file export tasks corresponding to the business service client are created according to the data volume and the file uniform resource locator;
[0100] Step S830: executing one or more file export tasks based on a preset task execution thread to obtain a business file corresponding to the business service client.
[0101] In an exemplary embodiment, one or more file export tasks corresponding to the business service client are created based on the data volume and the file uniform resource locator, which can be achieved in the following manner: determining the file parameters to be exported based on the file uniform resource locator, and determining the sub-data volume of each sub-parameter in the file parameters to be exported based on the data volume; generating one or more file export tasks based on the parameter name of the sub-parameter, the sub-data volume and the interface call type of the business service interface; placing the one or more file export tasks in a task cache queue, and executing each of the file export tasks in the task cache queue based on a preset task execution thread to obtain the business file corresponding to the business service client.
[0102] In an exemplary embodiment, determining whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client based on the file export request can be achieved in the following manner: parsing the file export request to obtain the file request method and file uniform resource locator of the business service client, and calling the business service interface of the business service client according to the file request method and file uniform resource locator; determining whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client based on the interface return information fed back by the business service interface.
[0103] In the above-mentioned file export method, on the one hand, since the file export request sent by the business service client can be received, and the business service interface can be determined according to the file export request whether it can be normally accessed and the data volume of the business file to be exported in the business service client; at the same time, when it is determined that the business service interface can be normally accessed, one or more file export tasks corresponding to the business service client are created according to the data volume and the file uniform resource locator, and one or more file export tasks are executed based on the preset task execution thread to obtain the business file corresponding to the business service client. Since there is no need to export all the business files at one time, the problem in the related technology that the file export server is burdened with a heavy burden due to the need to export all the business files in a certain server at one time is solved, the burden on the file export server is reduced, and the stability of the file export system is improved; on the other hand, since the business service can be used as a client, the decoupling between the business service and the file export service is achieved, the complexity of mutual calls between services is reduced, and the file export efficiency is improved.
[0104] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.
[0105] The exemplary embodiment of the present disclosure also provides a file export device. Specifically, refer to Fig. 9 As shown, the file export device may include a data size determination module 910, a file export task creation module 920, and a file export task execution module 930. Among them:
[0106] The data volume determination module 910 may be used to receive a file export request sent by a business service client, and determine whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request;
[0107] The file export task creation module 920 may be used to create one or more file export tasks corresponding to the business service client according to the data size and the file uniform resource locator when determining that the business service interface can be normally accessed;
[0108] The file export task execution module 930 may be used to execute one or more file export tasks based on a preset task execution thread to obtain a business file corresponding to the business service client.
[0109] In an exemplary embodiment of the present disclosure, one or more file export tasks corresponding to the business service client are created according to the data volume size and the file uniform resource locator, including: determining the file parameters to be exported according to the file uniform resource locator, and determining the sub-data volume of each sub-parameter in the file parameters to be exported according to the data volume size; generating one or more file export tasks according to the parameter name of the sub-parameter, the sub-data volume and the interface call type of the business service interface; placing the one or more file export tasks into a task cache queue, and executing each of the file export tasks in the task cache queue based on a preset task execution thread to obtain a business file corresponding to the business service client.
[0110] In an exemplary embodiment of the present disclosure, determining whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client is determined according to the file export request, including: parsing the file export request to obtain the file request method and file uniform resource locator of the business service client, and calling the business service interface of the business service client according to the file request method and file uniform resource locator; determining whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client according to the interface return information fed back by the business service interface.
[0111] The specific details of each module in the above-mentioned file export device have been described in detail in the corresponding file export method, so they will not be repeated here.
[0112] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0113] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0114] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
[0115] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0116] Refer to the following Fig.10 The electronic device 1000 according to this embodiment of the present disclosure is described. Fig.10 The electronic device 1000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0117] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include, but are not limited to: the at least one processing unit 1010, the at least one storage unit 1020, a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010), and a display unit 1040.
[0118] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1010 can perform the following steps: Figure 8 The step S810 shown in the figure is: receiving a file export request sent by a business service client, and determining whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client according to the file export request; step S820: when it is determined that the business service interface can be accessed normally, creating one or more file export tasks corresponding to the business service client according to the data volume and the file uniform resource locator; step S830: executing one or more file export tasks based on a preset task execution thread to obtain a business file corresponding to the business service client.
[0119] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202 , and may further include a read-only storage unit (ROM) 10203 .
[0120] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0121] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0122] The electronic device 1000 may also communicate with one or more external devices 1100 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or may communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0123] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0124] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.
[0125] According to the program product for implementing the above method in the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0126] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0127] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0128] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0129] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0130] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0131] Other embodiments of the present disclosure will be readily apparent to those skilled in the art after considering the specification and practicing the inventions invented herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not invented by the present disclosure. The specification and examples are to be considered merely exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.
Claims
1. A file export system, characterized in that: include: A business service client and a file export server, wherein the business service client is in communication connection with the file export server, wherein: The business service client is used to determine the file parameters to be exported, and generate a file export request according to the file parameters, the business service interface to be called to export the file parameters, and the interface call type of the business service interface; The file export server is used to receive a file export request sent by the business service client, and determine whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request; and When it is determined that the business service interface can be accessed normally, one or more file export tasks corresponding to the business service client are created according to the data size and the file uniform resource locator, and one or more file export tasks are executed based on a preset task execution thread to obtain a business file corresponding to the business service client.
2. The file export system according to claim 1, characterized in that: The file export server is a server with a large file export function, and the business service client is a micro server corresponding to the application.
3. The file export system according to claim 1, characterized in that: The business service client is also used to: Importing preset parameter annotations into the original business file in the business service client; wherein the parameter annotations include at least one of a time format parameter annotation, an enumeration value format parameter annotation, a table style parameter annotation, and a field parameter annotation to be exported; All class attribute information of all classes included in the original business file is obtained, and all class attribute information of all classes is parsed based on the imported parameter annotations to obtain file parameters that need to be exported.
4. The file export system according to claim 3, characterized in that: The business service client is also used to: Acquire the time attributes, enumeration value format attributes, table style attributes, and field attributes to be exported included in the original business file in the business service client; Importing the time format parameter annotation into the time class attribute, and importing the enumeration value format parameter annotation into the enumeration value format class attribute; Injecting the table style parameter annotation into the table style class attribute; wherein the table style parameter annotation includes at least one of a table width annotation, a table header height annotation, and a table content height annotation; Inject the field parameter annotation that needs to be exported into the field class attribute that needs to be exported; wherein the field parameter annotation includes the header alias of the data table that needs to be exported and / or the export order of the data table.
5. The file export system according to claim 3, characterized in that: The business service client is also used to: Parsing the timestamp data corresponding to the time attribute based on the time format parameter annotation to obtain the time parameter, and parsing the enumeration data corresponding to the enumeration value format attribute based on the enumeration value format parameter annotation to obtain the enumeration data parameter; Parsing the table style class attributes based on the table style parameter annotation to obtain the table class parameters, and parsing the data corresponding to the field class attributes to be exported based on the field parameter annotation to be exported to obtain the field parameters to be exported; The file parameters to be exported are generated according to the time parameters, enumeration data parameters, table parameters and field parameters to be exported.
6. The file export system according to claim 1, characterized in that: The file export server is also used for: Parsing the file export request to obtain the file request method and the file uniform resource locator of the business service client, and calling the business service interface of the business service client according to the file request method and the file uniform resource locator; According to the interface return information fed back by the business service interface, it is determined whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client.
7. The file export system according to claim 1, characterized in that: The file export server is also used for: Determining the file parameters to be exported according to the file uniform resource locator, and determining the sub-data volume of each sub-parameter in the file parameters to be exported according to the data volume; Generate one or more file export tasks according to the parameter name of the sub-parameter, the sub-data volume, and the interface call type of the business service interface; One or more file export tasks are placed in a task cache queue, and each of the file export tasks in the task cache queue is executed based on a preset task execution thread to obtain a business file corresponding to the business service client.
8. The file export system according to claim 7, characterized in that: The interface call type includes a hypertext transfer protocol type and / or a remote procedure call type; wherein the file export server is also used for: When the interface call type is a remote procedure call type, determining a Feign client corresponding to the business service client according to the business service name of the business service client; Based on the Feign client, the business file is obtained from the business service interface according to the data request mode of the business service interface.
9. The file export system according to claim 8, characterized in that: The file export server is also used for: According to the file uniform resource locator of the business service client, the data request mode of the business service interface is determined; wherein the data request mode includes a Get request mode and / or a Post request mode.
10. A file export method, characterized in that: Configured on the file export server, the file export method includes: Receive a file export request sent by a business service client, and determine whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request; When it is determined that the business service interface can be accessed normally, one or more file export tasks corresponding to the business service client are created according to the data volume and the file uniform resource locator; One or more file export tasks are executed based on a preset task execution thread to obtain a business file corresponding to the business service client.
11. The file export method according to claim 10, characterized in that: Creating one or more file export tasks corresponding to the business service client according to the data volume and the file uniform resource locator includes: Determining the file parameters to be exported according to the file uniform resource locator, and determining the sub-data volume of each sub-parameter in the file parameters to be exported according to the data volume; Generate one or more file export tasks according to the parameter name of the sub-parameter, the sub-data volume, and the interface call type of the business service interface; One or more file export tasks are placed in a task cache queue, and each of the file export tasks in the task cache queue is executed based on a preset task execution thread to obtain a business file corresponding to the business service client.
12. The file export method according to claim 10, characterized in that: Determining whether the business service interface can be normally accessed and the data volume of the business file to be exported in the business service client according to the file export request includes: Parsing the file export request to obtain the file request method and the file uniform resource locator of the business service client, and calling the business service interface of the business service client according to the file request method and the file uniform resource locator; According to the interface return information fed back by the business service interface, it is determined whether the business service interface can be accessed normally and the data volume of the business file that needs to be exported in the business service client.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the file export method according to any one of claims 10 to 12 is implemented.
14. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the file export method according to any one of claims 10 to 12 by executing the executable instructions.