Large file distributed processing method and device in reconciliation system
By adopting large file distributed processing methods in the channel reconciliation system of the third-party payment industry, the reconciliation files are logically sharded and processed in a multi-node cluster, solving the problems of low processing efficiency and insufficient stability of single nodes, realizing the Efficient file processing and stability improvement in low hardware configuration.
Patent Information
- Application Number
- CN202111102977.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-18
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2041-09-18
AI Technical Summary
The channel reconciliation system of the third-party payment industry has problems such as insufficient single-node stability and slow processing efficiency when processing large files, and relies on hardware configuration and memory databases, resulting in insufficient hardware and inefficiency.
Adopt large file distributed processing method to logically shard reconciliation files in file servers and multiple node clusters, and distribute and process file slices between multiple nodes through byte streaming, using numbering rules and sharding strategies. Ensure file integrity, set up monitoring modules and failover mechanisms to handle failure and timeout tasks.
Effectively avoids the problems of insufficient hardware and inefficiency in single-node processing, improves the efficiency of large files processing, solves the risk of single-point instability, and allows large files to be processed quickly in a low hardware configuration environment, supporting application microservices .
Smart Images
Figure CN114020716B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information technology, and in particular to a large file distributed processing method and device in a reconciliation system. Background Art
[0002] The channel reconciliation system in the third-party payment industry is mainly used to compare the third-party payment company with the two-link data (UnionPay, NetsUnion). The two-link transaction data is provided through files. When the transaction volume is large, the corresponding files are also relatively large. When the file size is larger than a certain proportion of the memory that can be processed by the application server, parsing the file will encounter a bottleneck.
[0003] At present, the channel reconciliation system in third-party payment mostly uses single-node processing. There are generally three ways to process large files. One is to increase the single-point memory, the second is to split the file and process it in sequence, and the third is to parse it one by one and put it into the memory database for processing. All three methods are single-node processing. Single-node processing of reconciliation files has two common disadvantages. One is that the single-node system is not stable enough. After a single node goes down, there is no other node to process the business. The second is that the single-node processing efficiency is slow and the single node can only process serially. The first method of increasing memory has another disadvantage, that is, the application is too dependent on hardware configuration. Another disadvantage of the third processing method is that it relies on middleware such as memory databases.
[0004] Therefore, how to perform distributed processing of large files in the reconciliation system to effectively avoid the drawbacks of insufficient hardware and slow processing efficiency in traditional solutions is an urgent problem to be solved by technical personnel in this field. Summary of the invention
[0005] The purpose of the present invention is to provide a large file distributed processing method and device in a reconciliation system to solve the problems raised in the above technical background.
[0006] To achieve the above object, the present invention adopts the following technical solutions:
[0007] The first aspect of the present application provides a large file distributed processing method in a reconciliation system, the method being applicable to a cluster including a file server and N nodes, each node corresponding to an application server, the file server providing a reconciliation file for at least one of the N nodes, N being a positive integer greater than or equal to 1, the method comprising:
[0008] Number the nodes where each application server in the cluster is located according to the preset numbering rules;
[0009] The file server logically segments at least one reconciliation file according to a preset segmentation strategy, and distributes the logically segmented files to the cache of the corresponding node in order;
[0010] Each node downloads the complete reconciliation file from the file server and stores the reconciliation file locally. Each node determines the corresponding file piece to be read according to the number of the current node, reads the file piece in the form of byte stream, and obtains the corresponding file piece data until the complete reconciliation file is successfully read by all nodes; wherein each file piece data read is stored in a continuous memory, including a first character at a starting position, an end character at an end position, and at least one line break character between the first character and the end character, and the line break character is used to trigger a data line break operation;
[0011] Acquiring the file piece data and performing data processing on the file piece data includes:
[0012] A, calculate the index position of the tail line break corresponding to the file piece data, the tail line break is the last line break contained in a file piece data; determine whether the end character of the file piece data is the tail line break, if so, remove the byte stream data after the tail line break; if not, read the reconciliation file again, read from the index position of the tail line break backward, end after reading the next line break, and then merge the byte stream data re-read into the byte stream data read by the previous segment;
[0013] B. Calculate the index position of the first line break corresponding to the file piece data, where the first line break is the first line break contained in a file piece data; determine whether the first character of the file piece data is the first line break, and if not, directly remove the byte stream data before the first line break.
[0014] Preferably, the preset numbering rule includes: the hash value of the IP address of the node where the application server is located modulo the number of nodes, which is used as the number Q of the node. n , that is, Q n =IP n %N; where Q n The number of the nth node in the cluster, IP n is the node IP value of the nth node in the cluster, and N is the number of nodes in the cluster.
[0015] Preferably, the numbering is implemented in at least one of the following ways:
[0016] a) Numbering before deployment on the application server;
[0017] b) After the application server is started, the corresponding node IP is obtained from the registration center and then numbered.
[0018] Preferably, the preset sharding strategy includes:
[0019] Get the memory size of the node, and take the preset ratio of the memory size as the size of the file slice;
[0020] Divide the size of the reconciliation file by the size of the file slice to get the number of shards corresponding to the reconciliation file. The number of shards corresponds to the number of nodes that need to process the reconciliation file.
[0021] When the required number of nodes is less than or equal to the number of actually deployed nodes, the file server directly allocates each node; when the required number of nodes is greater than the number of actually deployed nodes, the file server allocates each node in a round-robin manner.
[0022] Preferably, the index position of the trailing line break character is the size of the ending position of the file segment minus the difference between the trailing line break character and the ending position.
[0023] Preferably, the method also includes: after the file segment is read, the read information is saved in a file processing table of the database, and the information saved in the file processing table includes any one or more of the node IP, node number, index position of the beginning of the file segment, index position of the end of the file segment, index position of the trailing newline character, processing status, start time and end time.
[0024] Preferably, the method further comprises: setting a monitoring module, the monitoring module comprising a daemon thread for monitoring the file processing process of each node in the cluster, wherein:
[0025] Start the daemon thread to periodically scan the file processing process of each node. If all sharding tasks are processed normally, the current reconciliation file processing is completed; if a sharding task fails to be processed, or the processing time of the current node exceeds the preset time, the sharding task is judged to have failed to be processed, and the scheduled task will initiate reprocessing for the failed task.
[0026] More preferably, the method further includes: if a node processing fails, the number corresponding to the node is manually (or by a daemon thread) modified, and when the daemon thread scans and finds that the task of the current node matching the number is unsuccessful, a re-parse is initiated; after successful processing, the status information of the corresponding node is updated.
[0027] A second aspect of the present application provides a large file distributed processing device in a reconciliation system, the device being applicable to a cluster including a file server and N nodes, each node corresponding to an application server, the file server providing a reconciliation file for at least one of the N nodes, N being a positive integer greater than or equal to 1, the device comprising:
[0028] A node numbering unit is used to number the nodes where each application server in the cluster is located according to a preset numbering rule;
[0029] A logical sharding unit, used to logically shard a reconciliation file stored in the file server according to a preset sharding strategy, and to distribute the logically sharded file slices to the cache of the corresponding node in order;
[0030] A file slice reading unit, used for each node to download the complete reconciliation file from the file server and store the reconciliation file locally, each node determines the corresponding file slice to be read according to the number of the current node, reads the file slice in a byte stream manner, and obtains the corresponding file slice data until the complete reconciliation file is successfully read by all nodes; wherein each file slice data read is stored in a continuous memory, including a first character at a starting position, an end character at an end position, and at least one line break character between the first character and the end character, and the line break character is used to trigger a data line break operation;
[0031] The node data processing unit is used to obtain the file slice data and perform data processing on the file slice data so that there are no incomplete rows in the data read after logical sharding.
[0032] Preferably, the logical slicing unit includes:
[0033] A first calculation module is used to obtain the memory size of the node and take a preset ratio of the memory size as the size of the file slice;
[0034] A second calculation module is used to divide the size of the reconciliation file by the size of the file slice to obtain the number of slices corresponding to the reconciliation file, where the number of slices corresponds to the number of nodes that need to process the reconciliation file;
[0035] The node allocation module is used to determine that when the number of shards calculated by the second computing module is less than or equal to the number of nodes actually deployed, the file server directly allocates each node; when the number of shards calculated by the second computing module is greater than the number of nodes actually deployed, the file server allocates each node in a circular manner.
[0036] More preferably, the node data processing unit includes:
[0037] The first processing module is used to calculate the index position of the tail line break corresponding to the file piece data, where the tail line break is the last line break contained in a file piece data; determine whether the end character of the file piece data is the tail line break, and if so, remove the byte stream data after the tail line break; if not, read the reconciliation file again, read from the index position of the tail line break, and end after reading the next line break, and then merge the byte stream data re-read into the byte stream data read by the previous segment;
[0038] The second processing module is used to calculate the index position of the first line break corresponding to the file piece data, where the first line break is the first line break contained in a file piece data; determine whether the first character of the file piece data is the first line break, and if not, directly remove the byte stream data before the first line break.
[0039] Preferably, the device also includes: a monitoring module, the monitoring module includes a daemon thread, which is used to monitor the file processing process of each node in the cluster; wherein, the daemon thread is started to periodically scan the file processing process of each node, and if all sharding tasks are processed normally, the current reconciliation file processing is completed; if a sharding task fails to be processed, or the processing time of the current node exceeds the preset time, the sharding task is determined to have failed to be processed, and the failed task is reprocessed by the scheduled task.
[0040] More preferably, the device also includes: a failover processing module, which is used to manually modify the number corresponding to a node when processing of a node fails, and initiate re-parsing when the daemon thread scans and the task of the current node matching the number is unsuccessful; after successful processing, update the status information of the corresponding node.
[0041] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0042] The present application provides a method and device for distributed processing of large files in a reconciliation system. The reconciliation file processing adopts cluster deployment, and the reconciliation file is divided into several blocks (i.e., sharding, logical division, not physical division) through an algorithm, and the file content after logical sharding is distributed to the corresponding nodes in the cluster. Each node handles its own task respectively, effectively avoiding the drawbacks of insufficient hardware and slow processing efficiency in traditional solutions, so that the processing of large files can be quickly processed on virtual machines with low hardware configuration or cloud servers, which is convenient for the microservice of application programs. In addition, after distributed transformation, the potential risk of single point instability is completely solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:
[0044] Figure 1 It is a schematic diagram of the implementation mode of the cluster scenario of the large file distributed processing method of the present invention;
[0045] Figure 2 is a flow chart of the large file distributed processing method of the present invention;
[0046] Figure 3It is a schematic diagram of a file slicing implementation method of the large file distributed processing method of the present invention;
[0047] Figure 4 (a) to (c) are schematic diagrams of the byte stream processing process after shard reading of the large file distributed processing method of the present invention;
[0048] Figure 5 This is an example diagram of the effect after the byte stream of shard reading is processed;
[0049] Figure 6 It is a structural block diagram of the large file distributed processing device of the present invention. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0052] Embodiment 1:
[0053] Figure 1 It is a schematic diagram of an implementation method of the application cluster in the embodiment of the present application.
[0054] like Figure 1 As shown in Figure 1, a cluster may include a file server and multiple nodes, each of which corresponds to an application server. Figure 1 In the reconciliation application node 1, the reconciliation application node 2, the reconciliation application node 3, ..., the reconciliation application node N, N is a positive integer greater than or equal to 1. The file server is used to provide logically segmented reconciliation files to multiple nodes in the cluster.
[0055] Each node may be a device with a processing function, for example, any computing device known in the current technology, such as a server, a desktop computer, etc. The node may include a memory and a processor. The memory may be used to store program codes, such as an operating system and other application programs. The processor may be used to call the program code stored in the memory to implement the corresponding function of the node. The processor and memory included in the node may be implemented by a chip, which is not specifically limited here.
[0056] like Figure 2 As shown, the large file distributed processing method of the present application includes the following steps:
[0057] Step S1, The nodes where the application servers in the cluster are located are numbered according to the preset numbering rules and sharded by numbering.
[0058] Number the nodes where the application is deployed. The numbering rule is: the hash value of the IP address of the node where the application server is located modulo the number of nodes, which is used as the number of the node. n , that is, Q n =IP n %N; where Q n is the number of the nth node in the cluster, n≤N, IP n is the node IP value of the nth node in the cluster, and N is the number of nodes in the cluster.
[0059] When the reconciliation system is started, each node is numbered (0, 1, 2, ..., Q N ), there are many ways to implement numbering. You can do it before deploying the application, or you can do it after the application is started and obtain the corresponding IP from the registration center. If the reconciliation system deployed in the cluster is regarded as a complete system, then the numbering process can be regarded as sharding the reconciliation system, and the reading of the reconciliation file also needs to be processed by matching shards.
[0060] Step S2, The file server logically segments at least one reconciliation file according to a preset segmentation strategy, and distributes the logically segmented files to the cache of the corresponding node in sequence.
[0061] First, obtain the memory size of the node, and take the preset ratio of the memory size as the size of the file slice. Assume that the memory size of the node is m (unit: g), and the preset ratio is 1 / 2 of the memory size (this value can be adjusted appropriately according to the actual situation), then the size of the file read by the node each time is: m / 2.
[0062] Get the total size of the reconciliation file, divide the size of the reconciliation file by the size of the file slice, and get the number of slices corresponding to the reconciliation file. Assuming that the total size of the reconciliation file is v (unit: g), after slicing, the number of slices corresponding to the reconciliation file is: v / m / 2 (2v / m). The number of slices corresponds to the number of nodes that need to process the reconciliation file. When the number of nodes required is less than or equal to the number of nodes actually deployed, the file server directly allocates each node. When the number of nodes required is greater than the number of nodes actually deployed, the file server allocates each node in a cyclic manner.
[0063] During the reconciliation process, each node pulls the complete reconciliation file from the file server and stores it locally. When the application server reads the reconciliation file, it reads the reconciliation file in pieces according to the number of the current node. The reading is as follows:
[0064] When sharding, the file shard index needs to be converted into byte, which is defined as b, where b = m*1024*1024*1024;
[0065] The file slice read by node numbered 0 is: [0, (b / 2)*1];
[0066] The file slice read by node number 1 is: [(b / 2)*1+1, (b / 2)*2];
[0067] The file slice read by node number 2 is: [(b / 2)*2+1, (b / 2)*3];
[0068] …
[0069] The file slice read by the node numbered N is: [(b / 2)*Q N +1, (b / 2)*(Q N +1)];
[0070] When (b / 2)*(Q N +1)>=v*1024*1024*1024, take v.
[0071] For details on how to shard the reconciliation file, see Figure 3 Example diagram of .
[0072] Step S3, Each node reads the corresponding file piece through byte stream.
[0073] After the file is segmented, the corresponding file data can be read. The application reads the file through the byte stream. Therefore, after the file is logically segmented, it can be guaranteed that the application node will not repeat or miss the file after reading the segment. When all nodes are processed successfully, the processing of the complete file is also successful.
[0074] Step S4, After the file slice is read, data processing is performed on the acquired file slice data so that there are no incomplete rows in the read data.
[0075] When reading a file through a byte stream, a buffer (such as 1024 bytes) will be defined to read the file content. Since this application is logically sharded, when processing real reconciliation files, a complete business transaction is distinguished by a line break (other characters can be used to distinguish in special scenarios). There will be incomplete lines in the file read after logical sharding, and such data needs additional processing.
[0076] Each file piece data read is stored in a continuous memory, including a first character at a starting position, an end character at an end position and at least one line break character between the first character and the end character, and the line break character is used to trigger a data line break operation.
[0077] The file slice data is processed, specifically comprising the following steps:
[0078] A, record the index position of the tail line break corresponding to the file slice data, the index position of the tail line break is the size of the file slice end position minus the difference between the tail line break and the end position, the tail line break is the last line break contained in a file slice data. Determine whether the end character of the file slice data is the tail line break, if so, remove the byte stream data after the tail line break; if not, read the reconciliation file again, read from the index position of the tail line break, end after reading the next line break, and then merge the byte stream data re-read into the byte stream data read by the previous segment;
[0079] B, record the index position of the first line break corresponding to the file slice data, the first line break is the first line break contained in a file slice data. Determine whether the first character of the file slice data is the first line break. If not, directly remove the byte stream data before the first line break, and record the read information to the database for subsequent monitoring, fault transfer and other processing. After the processing is completed, the corresponding byte stream data can be converted into business object data for the remaining reconciliation processing.
[0080] Figure 4 (a)~ Figure 4 (c) are all example diagrams of the byte stream processing process after a file segment is read.
[0081] See also Figure 4As shown in (a), the byte stream data read by the segment includes a first character, a terminator, and three line breaks between the first character and the terminator. The three line breaks divide the byte stream data between the first character and the terminator into four lines of data. Among them, the first line of data is not a complete line, the second line of data and the third line of data are complete lines, and the fourth line of data is not a complete line. The first line break is located at the end of the first line of data, called the first line break, indicating the first line break of the byte stream data; the second line break is located at the end of the second line of data; the third line break is located at the end of the third line of data, and the fourth line of data is not a complete line, so the third line break is the last line break, indicating the last line break of the byte stream data.
[0082] According to step A of the above data processing, the terminator of the file slice is located at the fourth line of data and is not the tail newline. In order to ensure that a complete line of data is read, it is necessary to read the reconciliation file again, and read backward from the index position of the tail newline, and end after reading the next newline, that is, read the first newline position of the next file slice data adjacent to the file slice, and then merge the byte stream data re-read this time into the byte stream data read by the previous slice, so that the fourth line of data of the file slice is a complete line.
[0083] According to step B of the above data processing, the first character of the file piece is located in the first line of data and is not the first line break. In order to ensure that the complete line of data is read, the byte stream before the first line break needs to be removed.
[0084] See the effect diagram after byte stream processing Figure 5 .
[0085] Similarly, see Figure 4 As shown in (b), the byte stream data read by the segment includes a first character, a terminator, and two line feed characters between the first character and the terminator, and the two line feed characters divide the byte stream data between the first character and the terminator into three lines of data. Among them, the first line of data and the second line of data are complete lines, and the third line of data is not a complete line.
[0086] According to the above data processing steps A and B, the end character of the file slice is located at the third line of data, and is not the last line break. In order to ensure that the complete line of data is read, it is necessary to read the reconciliation file again, read from the index position of the last line break, and end after reading the next line break, that is, read the first line break position of the next file slice data adjacent to the file slice, and then merge the byte stream data re-read this time into the byte stream data read by the previous slice, so that the third line of data of the file slice is a complete line. The first line of data of the file slice is a complete line, and the first character is the first line break, which does not need to be processed.
[0087] Similarly, see Figure 4 As shown in (c), the byte stream data read by the segment includes a first character, a terminator, and two line feed characters between the first character and the terminator. The two line feed characters divide the byte stream data between the first character and the terminator into three lines of data. Among them, the first line of data is not a complete line, and the second line of data and the third line of data are complete lines. The first line feed character is located at the end of the first line of data, called the first line feed character, indicating the first line feed character of the byte stream data; the second line feed character is located at the end of the second line of data; the third line of data is a complete line, and the terminator is the last line feed character, that is, the terminator is the last line feed character.
[0088] According to the above data processing steps A and B, the end character of the file slice is the tail line break, and the byte stream data after the tail line break is directly removed. The first character of the file slice is not the first line break, and the byte stream data before the first line break is directly removed.
[0089] After the above-mentioned file segment is read, the read information needs to be saved in the file processing table of the database. The information saved in the file processing table includes node IP, node number, index position of the beginning of the file segment, index position of the end of the file segment, index position of the trailing newline character, processing status, start time and end time, etc.
[0090] In a preferred embodiment, the method further includes: monitoring and failover processing, specifically including:
[0091] File processing business monitoring: Start a daemon thread in the application and regularly scan the file processing table stored in the database. If all sharding tasks are processed normally, the current file processing is completed. If the processing fails, or the current node does not complete the processing within the preset time (for example, 30 minutes, which can be adjusted according to the current file processing time), the sharding task processing is judged to have failed. The failed tasks can be reprocessed by the scheduled task.
[0092] Failover processing: If a node fails to process and the current application is determined to be unavailable (other technical means such as health checks), the number corresponding to the node is manually (or by a daemon thread). When the daemon thread scans and finds that the task of the current node that matches the number is unsuccessful, it initiates a re-analysis; after successful processing, the status information of the corresponding node is updated. Here, the daemon thread can also directly modify the number. The daemon thread modification is system processing, and manual modification is manual intervention.
[0093] Embodiment 2:
[0094] See also Figure 6As shown, the present application also provides a large file distributed processing device in a reconciliation system, the device is applicable to a cluster including a file server and N nodes, each node corresponds to an application server, the file server provides a reconciliation file for at least one of the N nodes, N is a positive integer greater than or equal to 1, and the device includes:
[0095] The node numbering unit 100 is used to number the nodes where each application server in the cluster is located according to a preset numbering rule;
[0096] The logical sharding unit 200 is used to logically shard a reconciliation file stored in the file server according to a preset sharding strategy, and to distribute the logically sharded file slices to the cache of the corresponding node in sequence;
[0097] The file slice reading unit 300 is used for each node to download the complete reconciliation file from the file server and store the reconciliation file locally. Each node determines the corresponding file slice to be read according to the number of the current node, reads the file slice in the form of byte stream, and obtains the corresponding file slice data until the complete reconciliation file is successfully read by all nodes; wherein each file slice data read is stored in a continuous memory, including a first character at a starting position, an end character at an end position, and at least one line break character between the first character and the end character, and the line break character is used to trigger a data line break operation;
[0098] The node data processing unit 400 is used to obtain the file slice data and perform data processing on the file slice data so that there are no incomplete rows in the data read after logical sharding.
[0099] The above-mentioned logical slicing unit 200 includes:
[0100] The first calculation module 201 is used to obtain the memory size of the node and take a preset ratio of the memory size as the size of the file slice;
[0101] The second calculation module 202 is used to divide the size of the reconciliation file by the size of the file slice to obtain the number of slices corresponding to the reconciliation file, where the number of slices corresponds to the number of nodes that need to process the reconciliation file;
[0102] The node allocation module 203 is used to determine that when the number of shards calculated by the second calculation module 202 is less than or equal to the number of nodes actually deployed, the file server directly allocates each node; when the number of shards calculated by the second calculation module 202 is greater than the number of nodes actually deployed, the file server allocates each node in a circular manner.
[0103] The node data processing unit 400 includes:
[0104] The first processing module 401 is used to calculate the index position of the tail line break corresponding to the file piece data, where the tail line break is the last line break contained in a file piece data; determine whether the end character of the file piece data is the tail line break, and if so, remove the byte stream data after the tail line break; if not, read the reconciliation file again, read from the index position of the tail line break, and end after reading the next line break, and then merge the byte stream data re-read into the byte stream data read by the previous segment;
[0105] The second processing module 402 is used to calculate the index position of the first line break corresponding to the file piece data, where the first line break is the first line break contained in a file piece data; determine whether the first character of the file piece data is the first line break, and if not, directly remove the byte stream data before the first line break.
[0106] In a preferred embodiment, the device further includes: a monitoring module and a failover processing module; wherein, the monitoring module includes a daemon thread for monitoring the file processing process of each node in the cluster; wherein, the daemon thread is started to periodically scan the file processing process of each node, and if all sharding tasks are processed normally, the current reconciliation file processing is completed; if there is a sharding task processing failure, or the processing time of the current node exceeds the preset time, the sharding task processing is determined to have failed, and the scheduled task initiates reprocessing for the failed task. The failover processing module is used to manually (or by the daemon thread) modify the number corresponding to a node when the processing of a node fails, and initiate re-parsing when the daemon thread scans the current node with the number and fails; after successful processing, the status information of the corresponding node is updated.
[0107] In summary, the present application provides a method and device for distributed processing of large files in a reconciliation system. The reconciliation file processing adopts cluster deployment, and the reconciliation file is divided into several blocks (i.e., sharding, logical division, not physical division) through an algorithm, and the file content after logical sharding is distributed to the corresponding nodes in the cluster, and each node handles its own task respectively. After the channel reconciliation is transformed into a distributed one using the technical solution of the present application, the file processing efficiency is significantly improved, and the problem of insufficient memory on a single machine is effectively solved, so that the processing of large files can be quickly processed on a virtual machine with a lower hardware configuration or a cloud server, which is convenient for the microservice of the application. In addition, after the distributed transformation, the potential risk of single-point instability is completely solved.
[0108] The specific embodiments of the present invention are described in detail above, but they are only examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions made to the present invention are also within the scope of the present invention. Therefore, the equalization changes and modifications made without departing from the spirit and scope of the present invention should be included in the scope of the present invention.
Claims
1. A large file distributed processing method in a reconciliation system, the method being applicable to a cluster including a file server and N nodes, each node corresponding to an application server, the file server providing a reconciliation file for at least one of the N nodes, wherein N is a positive integer greater than or equal to 1; characterized in that: The method comprises: Number the nodes where each application server in the cluster is located according to the preset numbering rules; The file server logically segments at least one reconciliation file according to a preset segmentation strategy, and distributes the logically segmented files to the cache of the corresponding node in order; Each node downloads the complete reconciliation file from the file server and stores the reconciliation file locally. Each node determines the corresponding file piece to be read according to the number of the current node, reads the file piece in the form of byte stream, and obtains the corresponding file piece data until the complete reconciliation file is successfully read by all nodes; wherein each file piece data read is stored in a continuous memory, including a first character at a starting position, an end character at an end position, and at least one line break character between the first character and the end character, and the line break character is used to trigger a data line break operation; Acquiring the file piece data and performing data processing on the file piece data includes: A, calculate the index position of the tail line break corresponding to the file piece data, the tail line break is the last line break contained in a file piece data; determine whether the end character of the file piece data is the tail line break, if so, remove the byte stream data after the tail line break; if not, read the reconciliation file again, read from the index position of the tail line break backward, end after reading the next line break, and then merge the byte stream data re-read into the byte stream data read by the previous segment; B. Calculate the index position of the first line break corresponding to the file piece data, where the first line break is the first line break contained in a file piece data; determine whether the first character of the file piece data is the first line break, and if not, directly remove the byte stream data before the first line break.
2. The large file distributed processing method in the reconciliation system according to claim 1 is characterized in that: The preset numbering rule includes: the hash value of the node IP where the application server is located modulo the number of nodes, which is used as the number Q of the node n , that is, Q n =IP n %N; where Q n The number of the nth node in the cluster, IP n is the node IP value of the nth node in the cluster, and N is the number of nodes in the cluster.
3. The large file distributed processing method in the reconciliation system according to claim 1 is characterized in that: The preset sharding strategy includes: Get the memory size of the node, and take the preset ratio of the memory size as the size of the file slice; Divide the size of the reconciliation file by the size of the file slice to get the number of shards corresponding to the reconciliation file. The number of shards corresponds to the number of nodes that need to process the reconciliation file. When the required number of nodes is less than or equal to the number of actually deployed nodes, the file server directly allocates each node; when the required number of nodes is greater than the number of actually deployed nodes, the file server allocates each node in a round-robin manner.
4. The large file distributed processing method in the account reconciliation system according to claim 1 is characterized in that: The method further includes: setting a monitoring module, the monitoring module including a daemon thread, for monitoring the file processing process of each node in the cluster, wherein: Start the daemon thread to periodically scan the file processing process of each node. If all sharding tasks are processed normally, the current reconciliation file processing is completed; if a sharding task fails to be processed, or the processing time of the current node exceeds the preset time, the sharding task is judged to have failed to be processed, and the scheduled task will initiate reprocessing for the failed task.
5. The large file distributed processing method in the account reconciliation system according to claim 4 is characterized in that: The method also includes: if a node fails to be processed, manually or by a daemon thread, modifying the number corresponding to the node, and initiating a re-analysis when the daemon thread scans and finds that the task of the current node matching the number is unsuccessful; after successful processing, updating the status information of the corresponding node.
6. A large file distributed processing device in a reconciliation system, the device being applicable to a cluster including a file server and N nodes, each node corresponding to an application server, the file server providing a reconciliation file for at least one of the N nodes, wherein N is a positive integer greater than or equal to 1; characterized in that: The device comprises: A node numbering unit is used to number the nodes where each application server in the cluster is located according to a preset numbering rule; A logical sharding unit, used to logically shard a reconciliation file stored in the file server according to a preset sharding strategy, and to distribute the logically sharded file slices to the cache of the corresponding node in order; A file slice reading unit, used for each node to download the complete reconciliation file from the file server and store the reconciliation file locally, each node determines the corresponding file slice to be read according to the number of the current node, reads the file slice in a byte stream manner, and obtains the corresponding file slice data until the complete reconciliation file is successfully read by all nodes; wherein each file slice data read is stored in a continuous memory, including a first character at a starting position, an end character at an end position, and at least one line break character between the first character and the end character, and the line break character is used to trigger a data line break operation; The node data processing unit is used to obtain the file slice data and perform data processing on the file slice data so that there are no incomplete rows in the data read after logical sharding.
7. The large file distributed processing device in the account reconciliation system according to claim 6, characterized in that: The logical slicing unit includes: A first calculation module is used to obtain the memory size of the node and take a preset ratio of the memory size as the size of the file slice; A second calculation module is used to divide the size of the reconciliation file by the size of the file slice to obtain the number of slices corresponding to the reconciliation file, where the number of slices corresponds to the number of nodes that need to process the reconciliation file; The node allocation module is used to determine that when the number of shards calculated by the second computing module is less than or equal to the number of nodes actually deployed, the file server directly allocates each node; when the number of shards calculated by the second computing module is greater than the number of nodes actually deployed, the file server allocates each node in a circular manner.
8. The large file distributed processing device in the account reconciliation system according to claim 6, characterized in that: The node data processing unit comprises: The first processing module is used to calculate the index position of the tail line break corresponding to the file piece data, where the tail line break is the last line break contained in a file piece data; determine whether the end character of the file piece data is the tail line break, and if so, remove the byte stream data after the tail line break; if not, read the reconciliation file again, read from the index position of the tail line break, and end after reading the next line break, and then merge the byte stream data re-read into the byte stream data read by the previous segment; The second processing module is used to calculate the index position of the first line break corresponding to the file piece data, where the first line break is the first line break contained in a file piece data; determine whether the first character of the file piece data is the first line break, and if not, directly remove the byte stream data before the first line break.
9. The large file distributed processing device in the account reconciliation system according to claim 6, characterized in that: The device also includes: a monitoring module, the monitoring module includes a daemon thread, which is used to monitor the file processing process of each node in the cluster; wherein, the daemon thread is started to periodically scan the file processing process of each node, and if all sharding tasks are processed normally, the current reconciliation file processing is completed; if a sharding task fails to be processed, or the processing time of the current node exceeds a preset time, the sharding task is determined to have failed to be processed, and the failed task is reprocessed by the scheduled task.
10. The large file distributed processing device in the account reconciliation system according to claim 9, characterized in that: The device also includes: a failover processing module, which is used to manually or by a daemon thread modify the number corresponding to a node when processing of a node fails, and initiate re-analysis when the daemon thread scans and finds that the task of the current node matching the number is unsuccessful; after successful processing, the status information of the corresponding node is updated.