A method and system for fragmented timing of annual reports

Abstract

The application discloses a kind of annual report's fragmented timing execution method and system, the method includes: obtaining xxl-job instance total number and the subscript of current fragment;Obtain the enterprise code in annual report, and sort the enterprise code;Enter query request under current fragment, the query request includes fragment subscript;According to the fragment subscript, the annual report subtable under corresponding fragment is inquired, and the corresponding data under corresponding annual report subtable is periodically acquired.The method and system use xxl-job to execute fragmentation to the annual report data, respectively give different instance execution according to the type of annual report data, so that the code invasion of system can be reduced, so that the coupling degree of code is reduced, the updating efficiency of the annual report is improved.

Description

Technical Field

[0001] This invention relates to the field of big data, and in particular to a method and system for segmented and timed execution of annual reports. Background Technology

[0002] Currently, the system's annual report primarily utilizes the Quartz task scheduling middleware for automated task scheduling. By configuring tasks (specific business implementations) and triggers, a unified task scheduler handles the scheduling. The annual report task scheduling involves a large amount of data, integrating data from several large tables, including expense reports and personal expense reimbursements. In a multi-node online environment, Quartz can only schedule tasks through a single node, failing to effectively utilize the resources of other nodes. Furthermore, since Quartz tables and business tables reside in the same database, Quartz locks these tables during task execution, causing other tasks to be blocked and queued, impacting system stability. Summary of the Invention

[0003] One objective of this invention is to provide a method and system for timed execution of annual reports in segments. The method and system persistently persist annual report data to a database in segments on a timed basis, thereby improving the throughput of annual report data.

[0004] Another objective of this invention is to provide a method and system for sharding and timing the execution of annual reports. The method and system use XXL-job to shard the annual report data and assign it to different instances for execution according to the type of annual report data. This can reduce code intrusion into the system, reduce code coupling, and improve the update efficiency of the annual report.

[0005] Another objective of this invention is to provide a method and system for sharded and timed execution of annual reports. The method and system will count the number of instances on different nodes of the server and allocate the annual report data to instances through a routing strategy, thereby improving the execution effect of the annual report.

[0006] To achieve at least one of the above-mentioned objectives, the present invention further provides a method for segmented, timed execution of an annual report, the method comprising:

[0007] Get the total number of xxl-job instances and the index of the current shard;

[0008] Obtain the enterprise codes from the annual reports and sort the enterprise codes;

[0009] Enter a query request in the current shard, the query request including the shard index;

[0010] According to the shard subscript, the annual report subtable under the corresponding shard is queried, and corresponding data under the corresponding annual report subtable is obtained.

[0011] According to one of the preferred embodiments of the present application, an annotation @XxlJob is introduced in the annual report data. When the annotation exists in the annual report data, it is automatically discovered by the xxl-job scheduling center and is set as a shard broadcast by the routing of the xxl-job scheduling center, which is used to broadcast the annual report data to the corresponding instance.

[0012] According to another preferred embodiment of the present application, a matching algorithm is set, wherein the implementation process of the matching algorithm includes: obtaining the annual report data single quantity of an enterprise itself according to the enterprise code, setting a data single quantity threshold, if the obtained annual report data single quantity exceeds the data single quantity threshold, filtering the annual report data of the enterprise, and further performing subtabling on the annual report data corresponding to the remaining enterprise codes to generate a plurality of subtables.

[0013] According to another preferred embodiment of the present application, the matching algorithm includes: generating a corresponding subtable number according to the subscript of the instance node, wherein the subtable number contains the subscript content of the instance node, which is used to find the corresponding subtable according to the subscript node.

[0014] According to another preferred embodiment of the present application, the shard broadcast method includes: saving the expense data and the invoice data corresponding to the enterprise code in the corresponding subtable according to the result of the subtabling, wherein the code of the subtable includes the enterprise code, which is used to query the corresponding subtable code according to the enterprise code.

[0015] According to another preferred embodiment of the present application, the subtabling method further includes: setting a code quantity threshold, wherein after obtaining the number of enterprise codes according to the code quantity threshold, the total number of subtable codes not greater than the code quantity threshold is cut to generate, and the corresponding subtable total code number and the corresponding enterprise invoice and expense data are saved in the corresponding instance node.

[0016] According to another preferred embodiment of the present application, when the annual report data single quantity of an enterprise is greater than the data single quantity threshold, the data single corresponding to the enterprise code is scheduled and distributed according to the form capacity of each instance, and is preferentially configured to the subform under the instance subscript adjacent to the code.

[0017] According to another preferred embodiment of the present application, when the annual report data single quantity of an enterprise is greater than the data single quantity threshold, all instances need to be traversed in a loop during the subform querying process, until the current instance is the last instance, the pre-configured enterprise code is obtained through the last instance, and the last instance queries and obtains the corresponding enterprise invoice and expense data according to the enterprise code.

[0018] To achieve at least one of the above-mentioned objects, the present application further provides a system for performing a sharding timing of an annual report, which performs the above-mentioned method for performing a sharding timing of an annual report.

[0019] The present application provides a computer readable storage medium storing a computer program, which can be executed by a processor to perform the above-mentioned method for performing a sharding timing of an annual report. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A flowchart of a method for performing a sharding timing of an annual report is shown. DETAILED DESCRIPTION

[0021] The following description is provided to enable those skilled in the art to carry out the application. The preferred embodiments in the following description are only examples and other obvious modifications can be made by those skilled in the art. The basic principles defined in the following description can be applied to other embodiments, modifications, improvements, equivalents and other technical solutions without departing from the spirit and scope of the present application.

[0022] It can be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of one element can be one, and in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number.

[0023] Please refer to Figure 1 The present application discloses a method and system for performing a sharding timing of an annual report, wherein the method comprises the following parts: first, declare that the current job is an xxl-job, which can be discovered and scheduled by an xxl-job scheduling center after adding @xxl-job annotation to the task of the annual report, and further can be scheduled by the xxl-job scheduling center. The two instructions of ShardingUtil.getShardingVo().getIndex() and ShardingUtil.getShardingVo().getTotal() of the xxl-job scheduling center can respectively obtain the node sharding index corresponding to each instance of the xxl-job service and the number of instances.

[0024] Further, the enterprise code is obtained, wherein the enterprise code is preselected, the enterprise code code is an identification code of the enterprise in the xxl-job service, and the enterprise code can be used to find the annual report related data including but not limited to enterprise bills, expense data and the like saved by the enterprise in the xxl-job service.

[0025] After the enterprise code is obtained, the enterprise code is sorted, wherein the sorting rule is to sort the enterprise codes in sequence, for example, if the total number of enterprises in the system is 500, the corresponding enterprise code is configured as 001-500, each enterprise is allocated a corresponding enterprise code, and the enterprises are sorted from small to large according to the enterprise code, and after the sorting rule of the enterprise code is sorted, an enterprise code list is generated.

[0026] Further, since there are a large number of annual reports of different enterprises on the enterprise code list, annual reports of different enterprises are obtained from different enterprise systems, and therefore, in order to solve the problem of complex system docking codes, the present application adopts a xxl-job-based sharding broadcast mode to store and query annual report data. The method of sharding broadcast comprises the following steps: after the enterprise code is obtained, the total number of enterprise codes and the number of instances are used for table cutting. For example, if the total number of enterprise codes in the enterprise code table obtained in the xxl-job scheduling center is 500, the total number of instances is 10, and the maximum amount of annual report data carried by each instance is 100, the enterprise code table can be cut into 50 sub-tables, and the number of sub-tables is 10. The table cutting is cut according to the order of the enterprise code, so the enterprise code of the sub-table after the table cutting also satisfies the sorting rule. The sub-tables after the table cutting are allocated to the nodes of the corresponding instances, and the instance node subscript saved by the corresponding sub-table is recorded. For example, the subscript of the instance node is 0, and the current node has only one instance, so the node subscript 0 is allocated to the enterprise codes of the corresponding sub-tables 0-50, so that the subscript of the instance node can be obtained to query the corresponding sharding sub-table.

[0027] The xxl-job scheduling center obtains the annual report data of the corresponding enterprise through the interface provided by the different enterprises, identifies the corresponding enterprise, obtains the enterprise code in the xxl-job scheduling center and the total number of current instances, and stores the annual report data in the corresponding instance nodes in a sharding broadcast mode. By setting the timing task of the xxl-job scheduling center, the bill data, expense data and the like in the annual report data can be stored in the corresponding sub-tables.

[0028] It should be noted that, since the data form quantity of annual reports of different enterprises is different, the data form quantity of annual reports of relatively small enterprises is relatively small, so the present application will be divided into tables according to the data form carrying capacity of the service instance. According to the data form carrying capacity of the xx l-job service instance, a data form quantity threshold is set, after obtaining the data form quantity of the corresponding enterprise through the corresponding enterprise interface, it is judged whether the obtained data form quantity is not greater than the preset data form quantity threshold, if less than, the data form quantity of the annual report of the corresponding enterprise is retained, and the enterprise code of the corresponding enterprise is generated. At this time, the annual report forms of the enterprises greater than the data form quantity threshold are filtered out, so that the annual report forms of the enterprises less than the data form quantity threshold can be allocated to an instance for processing. Therefore, as long as the corresponding instance subscript is found, the corresponding subtable in the database of the node corresponding to the instance subscript is found directly according to the instance subscript, the corresponding subtable stores the corresponding enterprise code, and the required enterprise annual report related data is further obtained according to the enterprise code.

[0029] When the annual report data of the enterprise is greater than the preset data sheet quantity threshold, the xxl-job scheduling center is required to distribute the annual report data of the enterprise to different nodes for storage and perform data processing through different instances. In one preferred embodiment of the present application, when the annual report data of the enterprise is greater than the preset data sheet quantity threshold, the xxl-job scheduling center pre-configures different enterprise codes in the configuration file, such as configuring the enterprise codes less than the preset data sheet quantity threshold as continuous 001-500 codes, and configuring the enterprise codes greater than the preset data sheet quantity threshold as any value above 1000, which is used to distinguish enterprises with different data sheet quantities. Therefore, when the xxl-job task scheduling center can identify the special enterprise according to the pre-configured enterprise code, it further performs the traversal of all instances, obtains the number of all instances according to the ShardingUtil.getShardingVo().getTotal() instruction, and further obtains the index of the current instance according to ShardingUtil.getShardingVo().getIndex(). When traversing all instances starts, it further calculates ShardingUtil.getShardingVo().getIndex()-1, and the traversal calculation starts from the instance index 0. When ShardingUtil.getShardingVo().getIndex()-1 is equal to ShardingUtil.getShardingVo().getTotal(), it indicates that the current instance is the last instance, and the enterprise code greater than the data sheet quantity threshold is obtained through the last instance to obtain the corresponding single bill data, cost data and other related annual report data in each instance corresponding node. Thus, the problem of data balancing and search efficiency in the sharding process can be fully considered, and the coupling degree with different enterprise interfaces can be effectively reduced by establishing the form and instance search mode, so as to reduce the complexity of the code and improve the query efficiency.

[0030] The present application further limits the query quantity of the enterprise. Since the annual report of the enterprise contains a large amount of data, and the form needs to be associated with the query, the present application limits the maximum single query quantity of each enterprise form, such as setting the number of forms queried each time to 200. When there are more than 200 forms, the query operation of the annual report form needs to be performed in batches.

[0031] The application sets the route setting of the fragment broadcast, so that when there is a newly added enterprise annual form, the newly added form and the query of the newly added form can be generated respectively according to the number of instances. In the fragment task, the annotation @XxlJob is introduced, and the value field in the annotation is set, for example: demoJob. In the visual client of xxljob, when a new task is added, the JobHandler field in the form should be consistent with the value of the @XxlJob annotation, so that the task can be executed correctly.

[0032] In particular, according to the embodiments disclosed by the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network by a communication part, and / or installed from a detachable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection having one or more conductive segments, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries the computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to a wireless segment, a wire segment, an optical cable, an RF, etc., or any suitable combination thereof.

[0033] The computer program product of the present application can be a computer program product comprising a computer-readable medium bearing computer program code embodied therein for use with a computer. The computer program code can be code defining and / or implementing the present application. The computer program code can be organized into one or more computer program components, including but not limited to, program components for implementing the features of the present application. The one or more computer program components can each be a stand-alone computer program, or a plurality of computer program components can be combined to form a stand-alone computer program. The one or more computer program components can be organized as a computer source program, a computer recursive program, or a computer interpretive program. The one or more computer program components can be organized for execution by a host computer or a microprocessor, etc. The one or more computer program components can be organized for execution by a computer or a processing element of a computer.

[0034] Those skilled in the art will understand that the application described above and illustrated in the accompanying drawings is presented by way of example only and is not intended to limit the application. The present application thus extends to any and all embodiments within the scope of the following claims.

Claims

1. A method for segmented, timed execution of an annual report, characterized in that, The method includes: Get the total number of xxl-job instances and the index of the current shard; Obtain the enterprise codes from the annual report and sort the enterprise codes, including: assigning enterprise codes with a data volume less than a threshold to a first numerical range, the first numerical range being a continuous segment of codes, and assigning enterprise codes with a data volume greater than the threshold to a second numerical range, the minimum value of the second numerical range being greater than the maximum value of the first numerical range; Enter a query request in the current shard, the query request including the shard index; The annual report sub-table under the corresponding sub-table is retrieved based on the sub-table index, and the corresponding data under the corresponding annual report sub-table is retrieved periodically. The method further includes: setting a matching algorithm, wherein the implementation process of the matching algorithm includes: obtaining the number of annual report data items of the enterprise itself according to the enterprise code, setting a data item threshold, if the obtained data item number is less than the preset data item threshold, then retaining the corresponding enterprise annual report data item number and generating the corresponding enterprise code, so that the annual report forms of enterprises with less than the data item threshold are all assigned to one instance for processing, if the obtained annual report data item number exceeds the data item threshold, then filtering the annual report data items of the enterprise, and further dividing the remaining annual report data corresponding to the enterprise code into multiple sub-tables; When the number of enterprise annual report data entries exceeds the threshold, the query process for the sub-form needs to iterate through all instances until the current instance is the last one. Then, the pre-configured enterprise code is obtained from the last instance. Based on this last instance, the corresponding enterprise documents and expense data are retrieved, including: Get the total number of instances for the current task and the index of the current instance, and determine whether the index of the current instance is equal to the total number of instances minus 1. If so, the current instance is considered the last instance, and this instance is responsible for obtaining the pre-configured enterprise code and coordinating the aggregation of the required annual report data from the nodes corresponding to all instances.

2. The method for segmented and timed execution of an annual report according to claim 1, characterized in that, The annotation @XxlJob is introduced into the annual report data. When the annotation exists in the annual report data, it is automatically discovered by the xxl-job scheduling center and the routing of the xxl-job scheduling center is set to fragmented broadcast, which is used to broadcast the annual report data to the corresponding instances in fragments.

3. The method for segmented and timed execution of an annual report according to claim 1, characterized in that, The matching algorithm includes: generating a corresponding sub-table code based on the index of the instance, wherein the sub-table code contains the index content of the instance, and is used to find the corresponding sub-table based on the index content.

4. The method for segmented and timed execution of an annual report according to claim 2, characterized in that, The segmented broadcast includes: based on the results of the table segmentation, storing the expense data and document data corresponding to the enterprise code in the corresponding sub-table, wherein the code of the sub-table includes the enterprise code, and is used to query the corresponding sub-table code based on the enterprise code.

5. The method for segmented and timed execution of an annual report according to claim 1, characterized in that, The table partitioning also includes setting a coding quantity threshold, wherein after obtaining the number of enterprise codes based on the coding quantity threshold, the total number of codes in the sub-tables is not greater than the coding quantity threshold, and the total number of codes in the corresponding sub-tables and the corresponding enterprise documents and expense data are saved in the corresponding instance nodes.

6. The method for segmented and timed execution of an annual report according to claim 1, characterized in that, When the number of data forms in an enterprise's annual report exceeds the data form threshold, the data forms corresponding to the enterprise code will be scheduled and allocated according to the form capacity of each instance, and will be preferentially allocated to the sub-forms under the adjacent code of the instance.

7. A segmented, timed execution system for annual reports, characterized in that, The system executes a segmented, timed execution method for an annual report as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor as described in any one of claims 1-6: a segmented timed execution method for an annual report.

Images