Parallel processing method and system for batch cancel-after-verification tasks
Through parallel processing methods and systems, batch verification data is extracted and verified, and conflicts are identified and handled, which solves the problems of inefficiency and data conflicts in traditional methods, and achieves efficient and accurate verification task processing.
Patent Information
- Application Number
- CN202410509970.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-26
- Publication Date
- 2025-01-07
AI Technical Summary
The traditional batch verification method is inefficient, prone to calculation errors due to human factors, and serial processing cannot fully utilize the data processing capabilities of the computer system, resulting in data conflicts and unreasonable resource allocation, affecting the accuracy and consistency of the verification data .
Provide a parallel processing method and system for batch verification tasks. By extracting query data from preset verification windows, generating transaction files and performing template verification, purchasing-settlement mapping authentication, identifying verification conflicts, and updating parallel processing modules , use the updated module for parallel processing to improve processing efficiency and accuracy.
It significantly improves the processing efficiency and accuracy of batch verification tasks, avoids data conflicts and unreasonable resource allocation, ensures real-time extraction, verification and authentication of verification data, and ensures data accuracy and consistency.
Smart Images
Figure CN119273017A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of high performance computing technology, specifically to the field of parallel processing technology, and in particular to a parallel processing method and system for batch write-off tasks. Background Art
[0002] With the rapid development of the automotive consumer service sector, the scale of online and offline business has continued to expand, and the processing efficiency and accuracy of batch write-off tasks have become key factors in corporate operations. In this scenario, traditional batch write-off methods have gradually exposed their limitations. Manual calculations are not only inefficient and consume a lot of human resources, but are also prone to calculation errors due to human factors. Computer systems that mainly use serial processing cannot fully utilize the data processing capabilities of computer systems, resulting in low processing efficiency. At the same time, improper data processing can easily lead to technical problems such as data conflicts and unreasonable resource allocation, further affecting the accuracy and consistency of write-off data. Summary of the invention
[0003] The present application provides a parallel processing method and system for batch write-off tasks, aiming to solve the technical problems of data conflicts, unreasonable resource allocation and low processing efficiency caused by improper parallel processing in batch write-off tasks.
[0004] In view of the above problems, the present application provides a parallel processing method and system for batch write-off tasks.
[0005] The first aspect disclosed in the present application provides a parallel processing method for batch write-off tasks, the method comprising: extracting multiple query data of a preset write-off window, wherein the multiple query data comprises multiple external write-off codes and multiple write-off services; importing files based on the multiple external write-off codes and the multiple write-off services to generate multiple transaction files, wherein the multiple transaction files comprise multiple file name identifiers, and the file name identifiers are unique; performing template verification on the multiple file name identifiers, and if the verification passes, performing field parsing on the multiple external write-off codes to generate multiple fund files; based on the multiple transaction files Perform procurement-settlement mapping authentication with the multiple funding files, and if the authentication is passed, generate multiple files to be written off, wherein the multiple files to be written off have multiple request time identifiers; perform writing-off conflicts on the multiple files to be written off according to the multiple request time identifiers, and generate multiple conflicting file clusters to be written off, wherein the multiple conflicting file clusters to be written off have multiple conflict times; perform resource configuration update on the parallel processing module based on the multiple conflicting file clusters to be written off and the multiple conflict times, and obtain an updated parallel processing module; and use the updated parallel processing module to perform parallel processing on the multiple conflicting file clusters to be written off.
[0006] Another aspect disclosed in the present application provides a parallel processing system for batch write-off tasks, the system comprising: a data extraction component, the data extraction component is used to extract multiple query data of a preset write-off window, wherein the multiple query data include multiple external write-off codes and multiple write-off services; a file import component, the file import component is used to import files based on the multiple external write-off codes and the multiple write-off services to generate multiple transaction files, wherein the multiple transaction files include multiple file name identifiers, and the file name identifiers are unique; a field parsing component, the field parsing component is used to perform template verification on the multiple file name identifiers, and if the verification passes, the multiple external write-off codes are subjected to field parsing to generate multiple fund files; a mapping authentication component, the mapping authentication component is used to perform template verification on the multiple file name identifiers, and if the verification passes, the multiple external write-off codes are subjected to field parsing to generate multiple fund files; A procurement-settlement mapping authentication is performed on the multiple transaction files and the multiple funding files. If the authentication is passed, multiple files to be written off are generated, wherein the multiple files to be written off have multiple request time identifiers; a conflict identification component, wherein the conflict identification component is used to perform write-off conflict identification on the multiple files to be written off according to the multiple request time identifiers, and generate multiple conflicting file clusters to be written off, wherein the multiple conflicting file clusters to be written off have multiple conflict times; a configuration update component, wherein the configuration update component is used to perform resource configuration update on the parallel processing module based on the multiple conflicting file clusters to be written off and the multiple conflict times, and obtain an updated parallel processing module; a parallel processing module, wherein the parallel processing module is used to perform parallel processing on the multiple conflicting file clusters to be written off using the updated parallel processing module.
[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:
[0008] The above-mentioned parallel processing method for batch write-off tasks extracts multiple query data from a preset write-off window, including external write-off codes and write-off services. Then, these data are used to import files to generate multiple transaction files containing unique file name identifiers. Subsequently, these file name identifiers are template-checked to ensure that the format is correct. Once the verification is passed, the external write-off code will be further parsed to generate a fund file. After that, the transaction file and the fund file are combined to perform mapping authentication of procurement and settlement to ensure the accuracy and consistency of the data, and multiple files to be written off are generated, each of which has a corresponding request time identifier. Then, the write-off conflict is identified according to the obtained request time identifier, and the conflicting files are grouped to form a conflict file cluster to be written off, and the conflict time of each file cluster is recorded. In order to process these conflicting files more effectively, the resource configuration of the parallel processing module is updated according to the conflicting file cluster and the conflict time. In this way, a more efficient parallel processing module is obtained. Finally, the updated parallel processing module is used to parallel process the conflicting file cluster to be written off, thereby improving the processing efficiency and accuracy of the write-off task.
[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0011] Figure 1 A flowchart of a parallel processing method for batch write-off tasks in one embodiment;
[0012] Figure 2 The following is a diagram of the architecture of a parallel processing system for batch write-off tasks in one embodiment.
[0013] Explanation of the reference numerals: data extraction component 1, file import component 2, field parsing component 3, mapping authentication component 4, conflict identification component 5, configuration update component 6, parallel processing component 7. DETAILED DESCRIPTION
[0014] The embodiments of the present application provide a method and system for parallel processing of batch write-off tasks to solve the technical problems of data conflicts, unreasonable resource allocation and low processing efficiency caused by improper parallel processing in batch write-off tasks.
[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0016] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.
[0017] Embodiment 1
[0018] like Figure 1 As shown, the present application provides a parallel processing method for batch write-off tasks, the method comprising:
[0019] Extracting multiple query data of a preset write-off window, wherein the multiple query data includes multiple external write-off codes and multiple write-off services;
[0020] As the business scale continues to expand, the number of batch write-off tasks has also increased dramatically. The traditional serial processing method cannot cope with the processing needs of large-scale tasks, resulting in low processing efficiency and failure to meet the requirements of efficient operation of enterprises. The parallel processing method can process multiple tasks at the same time, significantly improving the processing speed and helping enterprises to quickly complete the write-off work.
[0021] In the embodiment of the present application, the system terminal obtains multiple query data items related to write-off from the preset write-off window. These data items include external write-off code and write-off service. The external write-off code is important information for identifying and tracking the write-off operation, while the write-off service refers to the specific write-off business content. By extracting these query data, the necessary information and basis can be provided for the subsequent write-off processing.
[0022] Importing files based on the multiple external write-off codes and the multiple write-off services to generate multiple transaction files, wherein the multiple transaction files include multiple file name identifiers, and the file name identifiers are unique;
[0023] In one embodiment, based on multiple external write-off codes and write-off services, the system terminal imports relevant files to generate a series of transaction files. These transaction files all contain unique file name identifiers to ensure that each file can be accurately identified and distinguished. For example, when the write-off service is a maintenance cancellation, the store plus product name plus settlement price plus timestamp can be used as a unique identifier. This not only facilitates subsequent data processing and management, but also improves the efficiency and accuracy of the write-off work.
[0024] Performing template verification on the multiple file name identifiers, and if the verification passes, performing field parsing on the multiple external verification codes to generate multiple fund files;
[0025] In one embodiment, the system terminal performs template verification on the file name identifiers of the generated multiple transaction files, mainly to ensure that these file names meet the preset specifications and are valid transaction files. Only after the verification based on the preset template indicator matrix is passed, the system terminal will further process these transaction files. The processing method is to perform field parsing on the external write-off code, that is, to extract key information from it, and generate multiple fund files based on this information. The fund file records the flow of funds related to the write-off, which is crucial for the subsequent write-off processing.
[0026] Furthermore, the present application provides template verification, including:
[0027] Extracting a plurality of preset template indicator matrices based on the plurality of file name identifiers, wherein the plurality of preset template indicator matrices include store names, product names, and settlement prices;
[0028] Taking the plurality of preset template indicator matrices as indexes, respectively extracting indicators from the plurality of transaction files to generate a plurality of transaction indicator matrices;
[0029] The template is verified according to the plurality of preset template indicator matrices and the plurality of transaction indicator matrices. If the store name, the product name and the settlement price are all verified successfully, the verification is passed.
[0030] Preferably, the system terminal first clarifies the structure of the preset template indicator matrix and the key fields contained therein, such as store name, product name, settlement price, etc. These fields are pre-defined according to the needs and rules of the enterprise's write-off business. Subsequently, the system field traverses all generated file name identifiers, and finds the preset template indicator matrix corresponding to each file name identifier based on a one-to-one comparison. Afterwards, for each matched preset template indicator matrix, the system terminal extracts these successfully matched preset template indicator matrices. These, multiple preset template indicator matrices are obtained. These preset template indicator matrices include store name, product name, and settlement price. Subsequently, the system terminal uses these preset template indicator matrices as indexes to extract the relevant indicators in the transaction file one by one, and forms a transaction indicator matrix based on the structure of the preset template indicator matrix. Finally, the system terminal compares and verifies the preset template indicator matrix with the transaction indicator matrix to ensure that the length, format, etc. of information such as store name, product name and settlement price are matched correctly. Only when these key information are successfully verified, the system terminal considers that the verification is passed. This step is to ensure the accuracy and consistency of the transaction file and provide reliable data support for subsequent write-off processing.
[0031] Performing procurement-settlement mapping authentication based on the multiple transaction files and the multiple fund files, and if the authentication is passed, generating multiple files to be written off, wherein the multiple files to be written off have multiple request time identifiers;
[0032] In one embodiment, based on multiple transaction files and fund files, the system terminal performs procurement-settlement mapping authentication. This authentication process is mainly to ensure the data consistency and accuracy between the transaction files and the fund files. Only when the transaction data and the fund data can match each other in the procurement and settlement links and pass the size check, the authentication will be judged as passed. Once the authentication is passed, the system terminal will generate multiple files to be written off based on these files. These files to be written off not only contain all the key information of previous transactions and funds, but also will be assigned one or more request time identifiers, which are used to identify the generation time of the file or the time sequence of the request processing. In summary, this process is to match and verify the transaction and fund data, and generate files to be written off accordingly, so as to carry out subsequent write-off processing.
[0033] Further, the present application provides a procurement-settlement mapping authentication based on the multiple transaction files and the multiple funding files, and the method further includes:
[0034] Performing one-to-one mapping between the multiple transaction files and the multiple fund files to generate multiple transaction file-fund file mapping relationships;
[0035] Extracting the purchase amount and the settlement amount based on the multiple transaction file-fund file mapping relationships, and generating multiple purchase-settlement mapping relationships;
[0036] Preferably, the system terminal traverses each transaction file and fund file to find the key fields between them. For example, order number, product name, timestamp, etc. are used to determine the corresponding relationship between the files. By comparing these fields, the system terminal can establish a one-to-one mapping relationship between the transaction file and the fund file. Once a matching field is found, the system terminal will generate a mapping relationship record to associate the identifier of the transaction file with the corresponding fund file identifier. During the mapping process, if it is found that a transaction file or fund file does not have a matching identifier or field, or multiple matches occur, the system terminal will perform corresponding exception processing, including recording errors, skipping unmatched files, prompting the user to perform manual matching, etc. Repeat this process until all files are matched, and multiple transaction file-fund file mapping relationships are generated. These mapping relationships ensure the accurate correspondence between transaction data and fund data. Subsequently, the system terminal traverses each mapping relationship record. For each record, the identifiers of the corresponding transaction file and fund file are obtained. Afterwards, the identifiers of the acquired transaction file and fund file are parsed, multiple purchase amounts and settlement amounts are extracted, and multiple purchase-settlement mapping relationships are formed. These mapping relationships not only reflect the flow of funds between procurement and settlement, but also provide the necessary data basis for subsequent write-off work. In summary, this process is to establish a one-to-one correspondence between transaction documents and fund documents, and extract the corresponding procurement and settlement amounts for subsequent write-off processing.
[0037] Determine whether the purchase amount in the multiple purchase-settlement mapping relationships is less than or equal to the settlement amount. If so, the authentication is passed, and the multiple purchase-settlement mapping relationships are used as multiple files to be written off;
[0038] If not, the authentication fails.
[0039] Preferably, the system terminal checks the multiple purchase-settlement mapping relationships generated one by one. In this process, the purchase amount and the settlement amount in each mapping relationship are compared to ensure that the purchase amount does not exceed the settlement amount. If the purchase amount of all mapping relationships meets this condition, that is, the purchase amount is less than or equal to the settlement amount, then the system terminal will determine that the authentication is passed and use these mapping relationships as files to be written off. These files to be written off will be used in the subsequent write-off process. However, if it is found that the purchase amount of any mapping relationship is greater than the settlement amount, then the authentication is not passed. This is because there are data errors or anomalies in these mapping relationships, which require further verification and adjustment. In summary, this process is to verify the consistency of the purchase amount and the settlement amount to ensure that they meet the basic requirements for write-off.
[0040] Identifying the conflicts of the multiple files to be written off according to the multiple request time identifiers, and generating multiple conflicting file clusters to be written off, wherein the multiple conflicting file clusters to be written off have multiple conflicting times;
[0041] In one embodiment, the system terminal identifies the cancellation conflict based on the request time identifier of the file to be cancelled. This is mainly to determine which files to be cancelled may have time overlap or conflict during the cancellation process. If the request time identifiers of two or more files to be cancelled have overlapping parts, they will be divided into a conflicting file cluster to be cancelled and assigned a conflict time identifier. Repeat multiple times until there is no conflict time, and the remaining multiple request time identifiers are divided into non-conflicting internal nodes. The conflict time refers to the time point between the start and end of the two request time identifiers, which represents the time period when these files may conflict during the cancellation process. Through such processing, the system terminal can clearly identify which files to be cancelled need special attention and processing to avoid conflicts or errors during the cancellation process.
[0042] Furthermore, the present application provides that, according to the multiple request time identifiers, the multiple to-be-written files are identified for writing-off conflicts, and multiple conflicting to-be-written file clusters are generated, and the method further includes:
[0043] A request time identifier is randomly extracted from the multiple request time identifiers to construct a first conflict identification unit, and the request time identifiers that meet the conflict window among the multiple request time identifiers are identified by using the first conflict identification unit, and divided into a first conflict identification internal node, and the remaining multiple request time identifiers are divided into a first to-be-divided set;
[0044] Randomly extract a request time identifier from the first set to be divided again to construct a second conflict identification unit, use the second conflict identification unit to identify the request time identifiers that meet the conflict window in the first set to be divided, divide them into the second conflict identification internal node, and divide the remaining multiple request time identifiers into the second set to be divided;
[0045] After multiple divisions, a request time identifier is randomly extracted from the N-1th set to be divided to construct an Nth conflict identification unit, and the request time identifiers that meet the conflict window in the N-1th set to be divided are identified by the Nth conflict identification unit and divided into the Nth conflict identification internal node, and the remaining multiple request time identifiers are divided into non-conflicting internal nodes;
[0046] Preferably, the system terminal randomly selects a request time identifier from all request time identifiers, and constructs a first conflict identification unit based on the extracted request time identifier. The core of this unit is a preset conflict judgment logic, which defines how to judge whether other request time identifiers conflict with the basic request time identifier of the unit. The conflict judgment logic selects a time identifier and compares the remaining time identifiers based on this time identifier to judge whether there is a conflict time. The first conflict identification unit will contain information about this conflict judgment logic and the basic request time identifier. Subsequently, the remaining request time identifiers are looped, and in each loop, the system terminal uses the first conflict identification unit to check the current request time identifier to determine whether the time overlap is satisfied. The request time identifiers that meet the conditions will be divided into the first conflict identification internal node, and those that do not meet the conditions will be divided into the first set to be divided. Afterwards, the system terminal will randomly select a request time identifier from the set to be divided again to construct a second conflict identification unit. Similarly, this unit will check whether other request time identifiers in the set to be divided meet the conflict window conditions, and those that meet the conditions will be divided into the second conflict identification internal node, and the remaining ones will enter the second set to be divided. This process will be repeated multiple times, and each time a request time identifier will be randomly selected from the previous set to be divided to construct a new conflict identification unit until the last set to be divided is processed. The last conflict identification unit will divide the request time identifiers that meet the conditions into the corresponding conflict identification internal nodes, and the remaining request time identifiers that have not been divided into any conflict identification internal nodes are considered to be non-conflicting and will be divided into non-conflicting internal nodes. In this way, the system terminal can efficiently identify all the files to be written off with time conflicts and organize them into different conflicting files to be written off clusters for easy subsequent processing and analysis.
[0047] A cancellation conflict identification module is constructed based on the first conflict identification unit, the second conflict identification unit and the Nth conflict identification unit, and the cancellation conflict identification module is used to identify the multiple files to be cancelled to obtain the first conflict identification internal node, the second conflict identification internal node and the Nth conflict identification internal node, as well as the non-conflicting internal node.
[0048] Preferably, the system terminal integrates the functions and logics of the constructed first conflict identification unit, the second conflict identification unit, and up to the Nth conflict identification unit to form a unified write-off conflict identification module. This module will integrate the core algorithms and judgment rules of all conflict identification units, and can efficiently and accurately identify conflicts in the documents to be written off. When using this module for identification, it can automatically divide the documents to be written off into corresponding conflict identification internal nodes or non-conflicting internal nodes according to the judgment results of each conflict identification unit. Ultimately, the system terminal will obtain the first conflict identification internal node, the second conflict identification internal node, up to the Nth conflict identification internal node, and those non-conflicting internal nodes that have not been divided into any conflict nodes. This process not only improves the efficiency and accuracy of conflict identification, but also provides a clear classification and basis for subsequent processing.
[0049] Furthermore, the present application provides a plurality of conflicting file clusters to be cancelled, and the method further includes:
[0050] Respectively taking the multiple request time identifiers stored in the first conflict identification internal node, the second conflict identification internal node and the Nth conflict identification internal node as N conflict file clusters to be cancelled;
[0051] The M request time identifiers in the non-conflicting internal nodes are respectively used as M conflicting file clusters to be cancelled;
[0052] The N conflicting file clusters to be cancelled and the M conflicting file clusters to be cancelled are used as the multiple conflicting file clusters to be cancelled.
[0053] Optionally, the system terminal extracts N request time identifiers from each conflict identification internal node, and uses these request time identifiers as N conflict file clusters to be cancelled. These conflict file clusters to be cancelled contain files to be cancelled that overlap or conflict in time. Subsequently, in the same way, the system terminal extracts M conflict file clusters to be cancelled. Afterwards, these conflict file clusters to be cancelled based on the conflict identification internal nodes and the non-conflicting internal nodes are integrated to form a collection of multiple conflict file clusters to be cancelled. This collection provides a clear data basis for subsequent processing and analysis.
[0054] Based on the multiple conflicting file clusters to be written off and the multiple conflicting times, resource configuration of the parallel processing module is updated to obtain an updated parallel processing module;
[0055] In one embodiment, the system terminal updates the resource configuration of the parallel processing module based on multiple conflicting file clusters to be written off and multiple conflicting times. This process is to identify the resource consumption of multiple conflicting file clusters to be written off through the resource configuration network layer, and reallocate processing resources according to the identification results, for example, increase or decrease the number of processing units, adjust the priority of processing units, etc., to achieve the function of dynamically adjusting the processing capacity of the parallel processing module. Doing so can ensure that the parallel processing module can process these conflicting files to be written off more efficiently, avoiding insufficient resources or waste of resources. After the resource configuration is updated, the system terminal will obtain an updated parallel processing module, which will have a better ability to process conflicting files to be written off, thereby improving the efficiency and accuracy of the entire write-off process.
[0056] Furthermore, the present application provides updating resource configuration of a parallel processing module based on the multiple conflicting file clusters to be written off and the multiple conflicting times to obtain an updated parallel processing module, and the method further includes:
[0057] Based on the multiple conflict times, the multiple conflicting file clusters to be cancelled are serialized in ascending order to generate a sequence of conflicting file clusters to be cancelled;
[0058] Traversing the conflicting file cluster sequence to be cancelled to perform information entropy calculation and generate an information entropy data sequence;
[0059] Preferably, the system terminal sorts multiple conflicting file clusters to be cancelled according to the order of multiple conflict times, and generates a sequence of conflicting file clusters to be cancelled. This sequence ensures that the file clusters are arranged according to the early and late times of their conflict times, which is helpful for the subsequent analysis and processing of the conflict situation. Subsequently, the system terminal creates an empty information entropy data sequence for storing the information entropy value of each conflicting file cluster to be cancelled. Then, the sequence of conflicting file clusters to be cancelled sorted according to the conflict time is traversed. For each conflicting file cluster to be cancelled in the sequence, information entropy calculation is performed. For the conflicting file cluster to be cancelled currently traversed, the system terminal obtains all request time identifiers and related data contained therein. Based on these data, the information entropy of the conflicting file cluster to be cancelled is calculated. The calculation of information entropy is based on the probability distribution of data. The system terminal will count the occurrence frequencies of different request time identifiers in the file cluster and calculate the information entropy based on these frequencies. After calculating the information entropy value of the current conflicting file cluster to be cancelled, the system knows to store it in the previously initialized information entropy data sequence. After completing the information entropy calculation of the current conflicting file cluster to be written off, the system terminal continues to traverse the next file cluster in the sequence and repeats the above steps until the entire sequence is traversed. After traversing all conflicting file clusters to be written off, the system terminal obtains a complete information entropy data sequence, which contains the information entropy value corresponding to each file cluster. This sequence provides important data support for subsequent analysis and decision-making, which can help the system terminal better understand the conflict situation and take corresponding treatment measures.
[0060] Inputting the information entropy data in the information entropy data sequence into the resource configuration network layer in sequence to identify resource consumption, thereby generating a resource consumption data sequence;
[0061] The resource configuration of the parallel processing module is updated based on the resource consumption data sequence to generate the updated parallel processing module.
[0062] Preferably, the system terminal sequentially inputs each information entropy data in the information entropy data sequence into the resource configuration network layer. This network layer is pre-trained and is a model specifically used to analyze data and predict resource consumption. It can evaluate the amount of resources required to process these conflicting file clusters to be written off based on the input information entropy data. The resource configuration network layer processes each information entropy data and generates corresponding resource consumption data. These resource consumption data reflect the computing resources, storage resources and other types of resources required to process the corresponding conflicting file clusters to be written off. Subsequently, the system terminal updates the resource configuration of the parallel processing module based on these resource consumption data. This means that the system terminal will dynamically adjust the resource allocation in the parallel processing module according to the resource consumption of each conflicting file cluster to be written off. For example, for file clusters with large resource consumption, the system terminal will increase the number of processing units or improve the performance of the processing units; while for file clusters with small resource consumption, the resource allocation will be reduced to improve the utilization efficiency of resources. After such a resource configuration update, the system terminal will obtain an updated parallel processing module. This module has more reasonable resource allocation and can better adapt to the processing needs of different conflicting file clusters to be written off, thereby improving the efficiency and performance of the entire processing flow.
[0063] Furthermore, the present application provides a training resource configuration network layer, and the method also includes:
[0064] Acquire multiple sample information entropy data and multiple sample resource consumption data as training data sets;
[0065] Equally divide the training data set to obtain n groups of training data sets;
[0066] The framework constructed based on the feedforward neural network is supervised and trained using the n sets of training data sets in turn, and the model network parameters are updated according to the output results of the previous set of training until the output results meet the preset output accuracy, thereby generating the trained resource configuration network layer.
[0067] Optionally, the system terminal retrieves a historical database to obtain multiple sample information entropy data and corresponding sample resource consumption data to form a training data set. These information entropy data reflect the complexity and uncertainty of the data, while the resource consumption data represents the amount of resources required to process these data. Subsequently, the training data set is evenly divided to obtain n sets of training data sets. This is done to use different data subsets to train the resource configuration network layer multiple times to improve the generalization ability and stability of the resource configuration network layer. After that, a framework based on a feedforward neural network is initialized. This framework includes an input layer, a hidden layer, and an output layer, as well as connection weights and bias terms between each layer. The number of hidden layers and the number of neurons in each layer are set based on the complexity of the problem and the characteristics of the data. Then, the system terminal loads the first set of n sets of training data sets, which includes information entropy data of multiple samples and corresponding resource consumption data. The information entropy data in the first set of training data sets is then used as input and forward propagated through the feedforward neural network. The data passes from the input layer through the hidden layer and finally reaches the output layer to generate predicted resource consumption data. Further, the system terminal compares the predicted resource consumption data with the actual resource consumption data, and calculates the value of the loss function to measure the difference between the predicted value and the actual value. Subsequently, according to the value of the loss function, the system terminal uses the back propagation algorithm to calculate the gradient, that is, the derivative of the loss function with respect to the network parameters. The calculated gradient is then used to update the network parameters of the feedforward neural network, including weights and bias terms. This process aims to reduce the value of the loss function so that the prediction performance of the feedforward neural network is gradually optimized. After completing the training of the first set of data, the next set of training data sets is loaded, and the above process is repeated to perform a new round of training and parameter update. After that, the system terminal sequentially loads the remaining n-2 sets of training data sets, and repeats the process of forward propagation, loss calculation, back propagation, and parameter update. In each iteration, the feedforward neural network adjusts its network parameters according to the training results of the previous set of data to gradually improve the prediction accuracy. After each round of iteration, the system terminal evaluates the output accuracy of the feedforward neural network on the current training data set. If the output accuracy reaches or exceeds the preset accuracy threshold, the training process ends; otherwise, the next round of iteration continues. When the training process meets the termination condition, that is, reaches the preset accuracy, the system terminal saves the final network parameters and generates a trained resource configuration network layer. This network layer can accurately predict the amount of resources required to process the input information entropy data, providing strong support for subsequent resource configuration updates.
[0068] The update parallel processing module is used to perform parallel processing on the multiple conflicting file clusters to be cancelled.
[0069] In one embodiment, after obtaining the updated parallel processing module, the system terminal uses this module to process multiple conflicting file clusters to be written off in parallel. Parallel processing means processing multiple file clusters at the same time, which can significantly improve processing efficiency. The updated parallel processing module optimizes resource allocation for different types of conflicting file clusters to be written off based on the previous resource configuration update, and can better deal with complex situations and avoid resource bottlenecks or waste. Therefore, using this updated module, multiple conflicting file clusters to be written off can be processed more efficiently, speeding up the write-off process and improving overall work efficiency.
[0070] In summary, the embodiments of the present application have at least the following technical effects:
[0071] The embodiment of the present application extracts multiple query data of the preset write-off window, which includes external write-off code and write-off service. Then, the transaction file is generated by using these data, and the uniqueness of the file name identifier is ensured. Subsequently, the transaction file is template-checked to generate the corresponding fund file. After that, the purchase-settlement mapping authentication is performed, and the file to be written off is generated by one-to-one mapping of the transaction file and the fund file. In this step, it is determined whether the purchase amount is less than or equal to the settlement amount to determine whether the authentication is passed or not. Then, the write-off conflict is identified for the file to be written off. By analyzing the request time identifier, the file to be written off is identified as a conflicting file cluster to be written off for subsequent processing. Then, the resource configuration of the parallel processing module is updated according to the conflicting file cluster to be written off and the conflict time to improve the processing efficiency. Finally, the resource configuration is optimized through the resource configuration network layer, and the efficiency and accuracy of the batch write-off task are improved. These technical effects jointly solve the technical problems of data conflict, unreasonable resource allocation and low processing efficiency caused by improper parallel processing in the batch write-off task, and realize the real-time extraction, verification, analysis and authentication of the write-off data, thereby ensuring the accuracy and consistency of the data.
[0072] Embodiment 2
[0073] Based on the same inventive concept as the parallel processing method for batch write-off tasks in the aforementioned embodiment, Figure 2 As shown, the present application provides a parallel processing system for batch write-off tasks, the system comprising:
[0074] Data extraction component 1: the data extraction component 1 is used to extract multiple query data of a preset write-off window, wherein the multiple query data includes multiple external write-off codes and multiple write-off services;
[0075] File import component 2: the file import component 2 is used to import files based on the multiple external write-off codes and the multiple write-off services to generate multiple transaction files, wherein the multiple transaction files include multiple file name identifiers, and the file name identifiers are unique;
[0076] Field parsing component 3: the field parsing component 3 is used to perform template verification on the multiple file name identifiers. If the verification passes, field parsing is performed on the multiple external verification codes to generate multiple fund files;
[0077] Mapping authentication component 4: the mapping authentication component 4 is used to perform procurement-settlement mapping authentication based on the multiple transaction files and the multiple fund files, and if the authentication is passed, generate multiple to-be-written-off files, wherein the multiple to-be-written-off files have multiple request time identifiers;
[0078] Conflict identification component 5: the conflict identification component 5 is used to identify the write-off conflicts of the multiple files to be written off according to the multiple request time identifiers, and generate multiple conflict file clusters to be written off, wherein the multiple conflict file clusters to be written off have multiple conflict times;
[0079] Configuration update component 6: the configuration update component 6 is used to update the resource configuration of the parallel processing module based on the multiple conflicting file clusters to be written off and the multiple conflicting times, so as to obtain an updated parallel processing module;
[0080] Parallel processing component 7: the parallel processing module 7 is used to use the update parallel processing module to perform parallel processing on the multiple conflicting file clusters to be cancelled.
[0081] Furthermore, the field parsing component 3 is also used to execute the following method:
[0082] Extracting a plurality of preset template indicator matrices based on the plurality of file name identifiers, wherein the plurality of preset template indicator matrices include store names, product names, and settlement prices;
[0083] Taking the plurality of preset template indicator matrices as indexes, respectively extracting indicators from the plurality of transaction files to generate a plurality of transaction indicator matrices;
[0084] The template is verified according to the plurality of preset template indicator matrices and the plurality of transaction indicator matrices. If the store name, the product name and the settlement price are all verified successfully, the verification is passed.
[0085] Furthermore, the mapping authentication component 4 is also used to execute the following method:
[0086] Performing one-to-one mapping between the multiple transaction files and the multiple fund files to generate multiple transaction file-fund file mapping relationships;
[0087] Extracting the purchase amount and the settlement amount based on the multiple transaction file-fund file mapping relationships, and generating multiple purchase-settlement mapping relationships;
[0088] Determine whether the purchase amount in the multiple purchase-settlement mapping relationships is less than or equal to the settlement amount. If so, the authentication is passed, and the multiple purchase-settlement mapping relationships are used as multiple files to be written off;
[0089] If not, the authentication fails.
[0090] Furthermore, the conflict identification component 5 is also used to execute the following method:
[0091] A request time identifier is randomly extracted from the multiple request time identifiers to construct a first conflict identification unit, and the request time identifiers that meet the conflict window among the multiple request time identifiers are identified by using the first conflict identification unit, and divided into a first conflict identification internal node, and the remaining multiple request time identifiers are divided into a first to-be-divided set;
[0092] Randomly extract a request time identifier from the first set to be divided again to construct a second conflict identification unit, use the second conflict identification unit to identify the request time identifiers that meet the conflict window in the first set to be divided, divide them into the second conflict identification internal node, and divide the remaining multiple request time identifiers into the second set to be divided;
[0093] After multiple divisions, a request time identifier is randomly extracted from the N-1th set to be divided to construct an Nth conflict identification unit, and the request time identifiers that meet the conflict window in the N-1th set to be divided are identified by the Nth conflict identification unit and divided into the Nth conflict identification internal node, and the remaining multiple request time identifiers are divided into non-conflicting internal nodes;
[0094] A cancellation conflict identification module is constructed based on the first conflict identification unit, the second conflict identification unit and the Nth conflict identification unit, and the cancellation conflict identification module is used to identify the multiple files to be cancelled to obtain the first conflict identification internal node, the second conflict identification internal node and the Nth conflict identification internal node, as well as the non-conflicting internal node.
[0095] Furthermore, the conflict identification component 5 is also used to execute the following method:
[0096] Respectively taking the multiple request time identifiers stored in the first conflict identification internal node, the second conflict identification internal node and the Nth conflict identification internal node as N conflict file clusters to be cancelled;
[0097] The M request time identifiers in the non-conflicting internal nodes are respectively used as M conflicting file clusters to be cancelled;
[0098] The N conflicting file clusters to be cancelled and the M conflicting file clusters to be cancelled are used as the multiple conflicting file clusters to be cancelled.
[0099] Furthermore, the configuration update component 6 is also used to execute the following method:
[0100] Based on the multiple conflict times, the multiple conflicting file clusters to be cancelled are serialized in ascending order to generate a sequence of conflicting file clusters to be cancelled;
[0101] Traversing the conflicting file cluster sequence to be cancelled to perform information entropy calculation and generate an information entropy data sequence;
[0102] Inputting the information entropy data in the information entropy data sequence into the resource configuration network layer in sequence to identify resource consumption, thereby generating a resource consumption data sequence;
[0103] The resource configuration of the parallel processing module is updated based on the resource consumption data sequence to generate the updated parallel processing module.
[0104] Furthermore, the configuration update component 6 is also used to execute the following method:
[0105] Acquire multiple sample information entropy data and multiple sample resource consumption data as training data sets;
[0106] Equally divide the training data set to obtain n groups of training data sets;
[0107] The framework constructed based on the feedforward neural network is supervised and trained using the n sets of training data sets in turn, and the model network parameters are updated according to the output results of the previous set of training until the output results meet the preset output accuracy, thereby generating the trained resource configuration network layer.
[0108] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order and continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0109] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
[0110] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.
Claims
1. A parallel processing method for batch write-off tasks, characterized in that: The method comprises: Extracting multiple query data of a preset write-off window, wherein the multiple query data includes multiple external write-off codes and multiple write-off services; Importing files based on the multiple external write-off codes and the multiple write-off services to generate multiple transaction files, wherein the multiple transaction files include multiple file name identifiers, and the file name identifiers are unique; Performing template verification on the multiple file name identifiers, and if the verification passes, performing field parsing on the multiple external verification codes to generate multiple fund files; Performing procurement-settlement mapping authentication based on the multiple transaction files and the multiple fund files, and if the authentication is passed, generating multiple files to be written off, wherein the multiple files to be written off have multiple request time identifiers; Identifying the conflicts of the multiple files to be written off according to the multiple request time identifiers, and generating multiple conflicting file clusters to be written off, wherein the multiple conflicting file clusters to be written off have multiple conflicting times; Based on the multiple conflicting file clusters to be written off and the multiple conflicting times, resource configuration of the parallel processing module is updated to obtain an updated parallel processing module; The update parallel processing module is used to perform parallel processing on the multiple conflicting file clusters to be cancelled.
2. The method according to claim 1, characterized in that The method comprises: Extracting a plurality of preset template indicator matrices based on the plurality of file name identifiers, wherein the plurality of preset template indicator matrices include store names, product names, and settlement prices; Taking the plurality of preset template indicator matrices as indexes, respectively extracting indicators from the plurality of transaction files to generate a plurality of transaction indicator matrices; The template is verified according to the plurality of preset template indicator matrices and the plurality of transaction indicator matrices. If the store name, the product name and the settlement price are all verified successfully, the verification is passed.
3. The method according to claim 1, characterized in that Procurement-settlement mapping authentication is performed based on the multiple transaction files and the multiple funding files, the method comprising: Performing one-to-one mapping between the multiple transaction files and the multiple fund files to generate multiple transaction file-fund file mapping relationships; Extracting the purchase amount and the settlement amount based on the multiple transaction file-fund file mapping relationships, and generating multiple purchase-settlement mapping relationships; Determine whether the purchase amount in the multiple purchase-settlement mapping relationships is less than or equal to the settlement amount. If so, the authentication is passed, and the multiple purchase-settlement mapping relationships are used as multiple files to be written off; If not, the authentication fails.
4. The method according to claim 1, characterized in that Identifying the multiple to-be-written-off files for writing-off conflicts according to the multiple request time identifiers, and generating multiple conflicting to-be-written-off file clusters, the method comprising: A request time identifier is randomly extracted from the multiple request time identifiers to construct a first conflict identification unit, and the request time identifiers that meet the conflict window among the multiple request time identifiers are identified by using the first conflict identification unit, and divided into a first conflict identification internal node, and the remaining multiple request time identifiers are divided into a first to-be-divided set; A request time identifier is randomly extracted from the first set to be divided again to construct a second conflict identification unit, and the request time identifiers that meet the conflict window in the first set to be divided are identified by the second conflict identification unit, and are divided into the second conflict identification internal node, and the remaining multiple request time identifiers are divided into the second set to be divided; After multiple divisions, a request time identifier is randomly extracted from the N-1th set to be divided to construct an Nth conflict identification unit, and the request time identifiers that meet the conflict window in the N-1th set to be divided are identified by the Nth conflict identification unit and divided into the Nth conflict identification internal node, and the remaining multiple request time identifiers are divided into non-conflicting internal nodes; A cancellation conflict identification module is constructed based on the first conflict identification unit, the second conflict identification unit and the Nth conflict identification unit, and the cancellation conflict identification module is used to identify the multiple files to be cancelled to obtain the first conflict identification internal node, the second conflict identification internal node and the Nth conflict identification internal node, as well as the non-conflicting internal node.
5. The method according to claim 4, characterized in that Respectively taking the multiple request time identifiers stored in the first conflict identification internal node, the second conflict identification internal node and the Nth conflict identification internal node as N conflict file clusters to be cancelled; The M request time identifiers in the non-conflicting internal nodes are respectively used as M conflicting file clusters to be cancelled; The N conflicting file clusters to be cancelled and the M conflicting file clusters to be cancelled are used as the multiple conflicting file clusters to be cancelled.
6. The method according to claim 1, characterized in that Based on the multiple conflicting file clusters to be written off and the multiple conflicting times, resource configuration of the parallel processing module is updated to obtain an updated parallel processing module, the method comprising: Based on the multiple conflict times, the multiple conflicting file clusters to be cancelled are serialized in ascending order to generate a sequence of conflicting file clusters to be cancelled; Traversing the conflicting file cluster sequence to be cancelled to perform information entropy calculation and generate an information entropy data sequence; Inputting the information entropy data in the information entropy data sequence into the resource configuration network layer in sequence to identify resource consumption, thereby generating a resource consumption data sequence; The resource configuration of the parallel processing module is updated based on the resource consumption data sequence to generate the updated parallel processing module.
7. The method according to claim 6, characterized in that The method comprises: Acquire multiple sample information entropy data and multiple sample resource consumption data as training data sets; Equally divide the training data set to obtain n groups of training data sets; The framework constructed based on the feedforward neural network is supervised and trained using the n sets of training data sets in turn, and the model network parameters are updated according to the output results of the previous set of training until the output results meet the preset output accuracy, thereby generating the trained resource configuration network layer.
8. A parallel processing system for batch write-off tasks, characterized in that: For implementing any one of the parallel processing methods 1-7 of the batch write-off task, the system comprises: Data extraction component: extracting multiple query data of a preset write-off window, wherein the multiple query data include multiple external write-off codes and multiple write-off services; File import component: importing files based on the multiple external write-off codes and the multiple write-off services to generate multiple transaction files, wherein the multiple transaction files include multiple file name identifiers, and the file name identifiers are unique; Field parsing component: performs template verification on the multiple file name identifiers, and if the verification passes, performs field parsing on the multiple external verification codes to generate multiple fund files; A mapping authentication component: performing procurement-settlement mapping authentication based on the multiple transaction files and the multiple fund files, and if the authentication is passed, generating multiple to-be-written-off files, wherein the multiple to-be-written-off files have multiple request time identifiers; Conflict identification component: performing write-off conflict identification on the multiple files to be written off according to the multiple request time identifiers, and generating multiple conflicting file clusters to be written off, wherein the multiple conflicting file clusters to be written off have multiple conflict times; Configuration update component: based on the multiple conflicting file clusters to be written off and the multiple conflicting times, update the resource configuration of the parallel processing module to obtain an updated parallel processing module; Parallel processing component: using the update parallel processing module to perform parallel processing on the multiple conflicting file clusters to be cancelled.