A data processing method for financial accounting office equipment
Through real-time monitoring and dynamic optimization and adjustment mechanisms, the problem of inefficient processing of traditional financial data is solved, and the efficient, accurate and real-time processing of financial data is achieved.
Patent Information
- Application Number
- CN202411528305.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-04-01
- Estimated Expiration
- 2044-10-30
AI Technical Summary
Traditional financial data processing methods are inefficient, especially when the data volume is large and the data sources are diverse, it cannot meet the needs of real-time processing.
Through real-time monitoring and analysis of business status, perform sharding and batch processing, and dynamically allocate tasks and optimize equipment selection based on dynamic optimization and adjustment mechanisms to improve data processing efficiency.
It realizes dynamic adjustment and optimization of financial data processing, improves the efficiency and accuracy of data processing, meets real-time processing needs, and maximizes resource utilization.
Smart Images

Figure CN119046104B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of financial data processing, and in particular to a data processing method for financial accounting office equipment. Background Art
[0002] Financial data is extremely important to enterprises. Traditional financial data processing usually involves manual tagging, statistics and integration. This method of financial data is prone to errors, especially when facing large amounts of complex data from large enterprises. Professional financial accounting office equipment automatically collects, processes, analyzes and stores financial data through access to multiple interfaces to ensure the consistency and accuracy of data processing. Therefore, it is very necessary for enterprises to use financial accounting office equipment to process financial data.
[0003] Currently, financial accounting office equipment usually processes data based on fixed processing procedures. This processing method has limitations. When the data volume is large and the data sources are diverse, the fixed processing procedures often cannot process the data efficiently. For example, during business peak periods or when a large amount of transaction data needs to be processed in real time, the traditional sequential processing method is inefficient and cannot meet the needs of real-time processing. Summary of the invention
[0004] The object of the present invention is to provide a data processing method for financial accounting office equipment to solve the problems mentioned in the above background.
[0005] The object of the present invention can be achieved by the following technical solution: A data processing method for financial accounting office equipment, comprising the following steps:
[0006] Step 1: Analyze the current business status in real time to determine whether optimization and adjustment are needed. If necessary, execute Step 2.
[0007] Step 2: The financial data is fragmented and batched to obtain several batches of data, where each batch of data includes several data fragments arranged in sequence, except for the first-ranked data fragment, and each data fragment corresponds to a batch value;
[0008] Step 3: Dynamically optimize and adjust each batch of data based on the batch value. The specific adjustment steps are:
[0009] S101: Treat each batch of data as an independent task, thereby obtaining a number of independent tasks and batch values corresponding to each data segment in the independent task, and summing them up to obtain the batch value of the independent task; thereby obtaining the batch value of each independent task, and selecting the independent task with the largest batch value as the calibration task;
[0010] S102: Set each device to correspond to a line energy value, recorded as B; obtain the number of independent tasks to be processed of each device in real time, and record it as Y; retrieve the busy value FR of each device closest to the current time of the system in real time, and normalize it with the number of independent tasks to be processed Y and the line energy value B, and take their values, analyze the values to obtain the thread value YR, and the specific calculation formula is:
[0011]
[0012] Where λ1, λ2, and λ3 are respectively set weight constants; thus, the thread values of several devices can be obtained, and the device with the largest thread value is selected as the calibration device, and the calibration task is assigned to the calibration device, thereby increasing the number of independent tasks to be processed of the calibration device by one;
[0013] S103: Whenever the device completes an independent task, the number of independent tasks to be processed corresponding to it is reduced by one, and a processing record is added, wherein the processing record includes the processing start time and the processing end time; the cumulative processing record of the device is obtained, and the processing start time and the processing end time in each processing record are calculated by time difference to obtain a single processing time, and the single processing time corresponding to each processing record is calculated by average to obtain a line energy value, and it is updated to S102;
[0014] S104: Repeat S101 to S103 until all independent tasks are processed;
[0015] Step 4: After each batch of data is processed, the processing results of each device are merged according to their corresponding category labels.
[0016] Preferably, the specific steps for determining whether optimization adjustment is needed are as follows:
[0017] S201: The amount of data to be processed, the data processing speed, and the broadband occupancy rate of each device are obtained in real time and recorded as Li, Vi, and Ui respectively; wherein i=1,2,3,...,I, I is a positive integer, I is the total number of acquisition moments, and i is the sequence number of any acquisition moment; the amount of data to be processed Li, the data processing speed Vi, and the broadband occupancy rate Ui are normalized and their values are taken, and the values are calculated and analyzed in a formula to obtain the load value Fi; the specific calculation formula is:
[0018]
[0019] Among them, β1, β2 and β3 are the set weight constants respectively;
[0020] S202: constructing a two-dimensional rectangular coordinate system with time as the horizontal coordinate and load value as the vertical coordinate, inputting the load value into the coordinate axis according to its corresponding collection time, and recording the position of the load value in the coordinate axis as the load point, and connecting each load point in sequence with a smooth curve to obtain a curve graph of load value changing with time;
[0021] S203: Perform fine-grained analysis on the load state of the device according to the load value versus time curve to obtain a busy value; this can obtain the busy value of each device;
[0022] S204: Compare and analyze the busy value of each device with the set busy interval. When the busy value is greater than the set busy interval, the device corresponding to the busy value is recorded as a highly busy device; when the busy value is in the set busy interval, the device corresponding to the busy value is recorded as a moderately busy device; when the busy value is less than the set busy interval, the device corresponding to the busy value is recorded as a low busy device; count the number of highly busy devices, moderately busy devices and low busy devices respectively, and record them as A1, A2 and A3 respectively; if A3≥A1+A2, no optimization adjustment is required; otherwise, execute Step 2.
[0023] Preferably, the specific steps of performing fine-grained analysis on the load status of the device according to the load value versus time curve are as follows:
[0024] Draw a tangent line at each load point, and calculate the derivative of the load at each load point by taking the derivative of the curve. Sum the load derivatives greater than zero to get the load increase degree, which is recorded as R1, and sum the load derivatives less than zero and take the absolute value to get the load reduction degree, which is recorded as R2. Normalize the load value Fi, load increase degree R1, and load reduction degree R2 of the equipment at each collection time and take their values. Analyze the values to get the busy value FR. The specific calculation formula is:
[0025]
[0026] Among them, α1 and α2 are the set weight constants respectively.
[0027] Preferably, the specific steps of sharding are as follows:
[0028] The financial data is segmented according to a specific byte size to obtain a number of data segments, keywords of each data segment are extracted, and the importance score of each keyword in the data segment is identified using TF-IDF; thereby, the keywords in each data segment and the importance scores corresponding to the keywords can be obtained;
[0029] Suppose there are several categories, each category corresponds to a label, and each category label corresponds to several keywords;
[0030] Extract the keywords of the data segment and compare them with the keywords corresponding to the category label to obtain the intersection keywords and union keywords, and count the number of intersection keywords and union keywords, which are recorded as N1 and N2 respectively;
[0031] Retrieve the importance scores of each intersection keyword in the data segment, and sum them up to obtain the total importance score; compare and analyze the importance scores of each intersection keyword with the set score range to obtain the cumulative number of high intersection, medium intersection and low intersection, and record them as D1, D2 and D3 respectively;
[0032] The number of intersection keywords N1, the number of union keywords N2, the cumulative number of high intersections D1, the cumulative number of medium intersections D2, the cumulative number of low intersections D3 and the total important score D4 are normalized and their values are taken. The numerical values are calculated and analyzed in a formula to obtain the score ND. The specific calculation formula is:
[0033]
[0034] Among them, η1, η2, and η3 are the set weight constants respectively, and η1>η2>η3>1; thus, the classification value between the data fragment and each type of target tag can be obtained, and the class tag with the largest classification value is selected as the target tag of the data fragment, and the data fragment is classified into the target tag; thus, all data fragments can be classified into each type of target tag.
[0035] Preferably, the specific steps of batch processing are as follows:
[0036] S501: Select any category tag and the data segments it includes, and use the data segment whose timestamp is farthest from the current system as the calibration segment, and use the other data segments as the comparison segments; obtain the overlap value by analyzing the overlap degree between the calibration segment and the comparison segment, thereby obtaining the overlap value between each calibration segment and each comparison segment;
[0037] S502: Calculate the time difference between the timestamp corresponding to each comparison segment and the current time of the system to obtain the interval time, and record it as T; it should be noted that the data segment with a larger interval time should be processed first; the interval time T and the coincidence value MC are normalized and their values are taken, and the numerical values are analyzed to obtain the batch value TM of the comparison segment. The specific calculation formula is:
[0038]
[0039] Among them, α1 and α2 are the set weight constants respectively;
[0040] S503: starting with the calibration segment, sorting the comparison segments in descending order according to their corresponding batch values; starting with the calibration segment, intercepting a set amount of data segments as the same batch of data;
[0041] S504: Repeat the above S501 to S503 until all the data segments in the category tag are processed in batches; thus, the data segments in each category tag can be processed in batches.
[0042] Preferably, the specific steps of analyzing the degree of overlap between the calibration fragment and the comparison fragment are as follows:
[0043] Retrieve the keywords between the calibration data segment and the comparison data segment and the corresponding importance scores of the keywords, compare the keywords of the two to obtain the intersection keywords and the union keywords, and count the number of intersection keywords and union keywords, which are recorded as M1 and M2 respectively; then calculate the average of the importance scores of the union keywords in the calibration segment and the comparison segment to obtain the importance value, thereby obtaining the importance value of each intersection keyword, and sum them up to obtain the total importance value recorded as C4;
[0044] Compare and analyze the importance value of the intersection keyword with the set importance interval to obtain the cumulative number of heavy correlation, medium correlation and light correlation, and record them as C1, C2 and C3 respectively;
[0045] The intersection keyword M1, the union key number M2, the cumulative number of heavily correlated times C1, the cumulative number of moderately correlated times C2, the cumulative number of lightly correlated times C3 and the important total value C4 are normalized and their values are taken, and the values are calculated by formula to obtain the overlap value MC between the calibration segment and the comparison segment; the specific calculation formula is:
[0046]
[0047] Among them, β1, β2, and β3 are set weight constants respectively, and β1>β2>β3>1.
[0048] Beneficial effects of the present invention:
[0049] 1. By real-time monitoring of the amount of data to be processed, data processing speed and bandwidth occupancy of the equipment, and comprehensive analysis and calculation of the load value of each device, and analyzing the load change trend over time, the equipment status can be accurately judged to decide whether to trigger optimization adjustment;
[0050] 2. By slicing financial data according to specific byte sizes and using the TF-IDF algorithm to identify keywords in each data segment and their corresponding importance scores, it is possible to accurately determine the relevance of data segments to various target tags, ensure accurate data classification, reduce the probability of misclassification, simplify the process of manual intervention, and improve the efficiency and accuracy of data classification;
[0051] 3. By comprehensively analyzing and sorting the timestamp differences and coincidence values of the compared fragments, and assigning reasonable processing priorities to different data fragments, frequent context switching can be avoided, thereby reducing the delay caused by irrelevant data processing and improving the overall data processing efficiency; at the same time, the situation of processing data fragments too early or too late is avoided, thereby improving the balance and efficiency of the overall processing;
[0052] 4. Through continuous monitoring and adjustment of equipment load and performance, and dynamic adjustment of task allocation accordingly, resource utilization is maximized, processing bottlenecks and task backlogs are avoided, tasks can be adaptively allocated, equipment selection is optimized, and ultimately the data processing capacity and stability of the entire system are improved, realizing a dynamic adjustment and optimization mechanism for financial data processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The present invention will be further described below in conjunction with the accompanying drawings.
[0054] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0056] See also Figure 1 As shown, the present invention is a data processing method for financial accounting office equipment, comprising the following steps:
[0057] Step 1: Analyze the current business status in real time to determine whether optimization and adjustment are needed; the specific judgment process is as follows:
[0058] The amount of data to be processed, the data processing speed (i.e., the time required for a single data task from reception to completion) and the broadband occupancy rate of each device are obtained in real time, and are recorded as Li, Vi and Ui respectively; where i=1,2,3,...,I, I is a positive integer, I is the total number of acquisition moments, and i is the sequence number of any acquisition moment; it should be noted that the broadband occupancy rate refers to the ratio between the current actually used network bandwidth and the total available bandwidth of the network connection. When the broadband occupancy rate is large, such as 80% or more, it means that the network broadband is close to saturation, which means that the network transmission task is close to the carrying capacity of the system, and there is a risk of congestion and delay;
[0059] The amount of data to be processed Li, the data processing speed Vi and the broadband occupancy rate Ui are normalized and their values are taken, and the values are calculated and analyzed by formula to obtain the load value Fi; the specific calculation formula is:
[0060]
[0061] Wherein β1, β2 and β3 are respectively set weight constants, and the values of the weight constants are set by the personnel in this field according to the actual situation. It can be seen from the formula that when the amount of data to be processed by the device is larger, the data processing speed is smaller, and the bandwidth occupancy rate is larger, it means that the load of the device processing data is larger, and the load value is larger;
[0062] A two-dimensional rectangular coordinate system is constructed with time as the horizontal coordinate and load value as the vertical coordinate. The load value is input into the coordinate axis according to its corresponding collection time, and the position of the load value in the coordinate axis is recorded as the load point. A smooth curve is used to connect each load point in sequence to obtain a curve graph of the load value changing with time; a tangent line is drawn at each load point, and the tangent line is derived to obtain the load derivative at each load point. It should be noted that when the load derivative is greater than zero, it indicates that the load value at the load point is in an increasing trend; when the load derivative is less than zero, it indicates that the load value at the load point is in a decreasing trend; the load derivatives greater than zero are summed to obtain a load increase degree recorded as R1, and the load derivatives less than zero are summed and the absolute value is taken to obtain a load reduction degree recorded as R2; the load value Fi, load increase degree R1, and load reduction degree R2 of the equipment at each collection time are normalized and their values are taken, and the busy value FR is obtained by numerical analysis; the specific calculation formula is:
[0063]
[0064] Among them, α1 and α2 are respectively set weight constants. It can be seen from the formula that when the load value has a greater increasing trend and a smaller decreasing trend, and the larger the load value, it means that the current device is in the peak period of business; thus, the busy value of each device can be obtained, and it is compared and analyzed with the set busy interval. When the busy value is greater than the set busy interval, the device corresponding to the busy value is recorded as a highly busy device; when the busy value is in the set busy interval, the device corresponding to the busy value is recorded as a moderately busy device; when the busy value is less than the set busy interval, the device corresponding to the busy value is recorded as a low busy device; the number of highly busy devices, moderately busy devices and low busy devices is counted respectively, and they are recorded as A1, A2 and A3 respectively; if A3≥A1+A2, it means that the device currently processing financial data can fully accommodate the processing of financial data and maintain normal work, and no optimization adjustment is required; otherwise, execute Step 2;
[0065] By real-time monitoring of the amount of data to be processed, data processing speed, and broadband occupancy of the device, and by comprehensively analyzing and calculating the load value of each device and analyzing the load change trend over time, it is possible to accurately judge the device status and decide whether to trigger optimization adjustments.
[0066] Step 2: By sharding and batch processing financial data, conflicts during data integration of a single device can be reduced and the efficiency of financial data processing can be improved; specifically:
[0067] Sharding:
[0068] The financial data is segmented according to a specific byte size (the byte size here is set by the personnel in this field according to the actual situation) to obtain a number of data segments, the keywords of each data segment are extracted, and the importance score of each keyword in the data segment is identified by using TF-IDF (term frequency-inverse document frequency); thereby, the keywords in each data segment and the importance scores corresponding to the keywords can be obtained;
[0069] It is assumed that there are several categories, each of which corresponds to a label. For example, specific category labels include accounts receivable, cost, expenditure and net profit, etc. Each category label corresponds to several keywords. For example, the keywords corresponding to the category label "cost" include accounts receivable, uncollected accounts, labor cost, generation cost, inventory cost, fixed cost, etc. It should be noted that specific category labels and keywords are set by personnel in this field according to actual needs;
[0070] Extract the keywords of the data segment and compare them with the keywords corresponding to the category label to obtain the intersection keywords and union keywords, and count the number of intersection keywords and union keywords respectively, which are recorded as N1 and N2 respectively; it should be noted that the intersection keywords refer to the common keywords between the two, and the union keywords refer to all non-repeated keywords between the two;
[0071] The importance scores of each intersection keyword in the data segment are retrieved, and the sum of the scores is calculated to obtain the total importance value recorded as D4; the importance scores of each intersection keyword are compared and analyzed with the set score interval. When the importance score is greater than the set upper limit of the score interval, a high degree of intersection is accumulated; when the importance score is in the set score interval, a medium degree of intersection is accumulated; when the importance score is greater than the set upper limit of the score interval, a low degree of intersection is accumulated; the cumulative number of high degree of intersection, medium degree of intersection and low degree of intersection are counted respectively, and they are recorded as D1, D2 and D3 respectively; the number of intersection keywords N1, the number of union keywords N2, the cumulative number of high degree of intersection D1, the cumulative number of medium degree of intersection D2, the cumulative number of low degree of intersection D3 and the total importance score D4 are normalized and their values are taken, and the numerical values are analyzed and calculated in a formula to obtain the sub-standard value ND. The specific calculation formula is:
[0072]
[0073] Among them, η1, η2, and η3 are respectively the set weight constants, and η1>η2>η3>1; It can be seen from the formula that when the number of intersection keywords is greater and the corresponding importance score is greater, it means that the data segment has a greater correlation with the target tag of the category, and the score value is greater; thus, the score value between the data segment and each target tag can be obtained, and the target tag with the largest score value is selected as the target tag of the data segment, and the data segment is classified into the target tag; thus, all data segments can be classified into each target tag;
[0074] By segmenting financial data into specific byte sizes and using the TF-IDF algorithm to identify keywords in each data segment and their corresponding importance scores, we can accurately determine the correlation between data segments and various target tags, ensure accurate data classification, reduce the probability of misclassification, simplify the process of manual intervention, and improve the efficiency and accuracy of data classification.
[0075] Batch Processing:
[0076] Step 1: Select any category tag and the data segments it includes, and use the data segment with the farthest timestamp from the current system as the calibration segment, and use the other data segments as the comparison segments; retrieve the keywords between the calibration data segment and the comparison data segment and the importance scores corresponding to the keywords, compare the keywords of the two to obtain the intersection keywords and the union keywords, and count the number of intersection keywords and union keywords, which are recorded as M1 and M2 respectively; then calculate the average of the importance scores of the union keywords in the calibration segment and the comparison segment to obtain the importance value, thereby obtaining the importance value of each intersection keyword, and sum them up to obtain the total importance value recorded as C4; compare the importance value of the intersection keyword with the set importance region When the importance value is greater than the set upper limit of the importance interval, a heavy correlation is accumulated; when the importance value is in the upper and middle limits of the important interval, a moderate correlation is accumulated; when the importance value is greater than the lower limit of the important interval, a slight correlation is accumulated; the cumulative number of heavy correlation, moderate correlation and slight correlation is counted respectively, and they are recorded as C1, C2 and C3 respectively; the intersection keyword M1, the union key number M2, the cumulative number of heavy correlation C1, the cumulative number of moderate correlation C2, the cumulative number of slight correlation C3 and the total importance value C4 are normalized and their values are taken, and the values are calculated by formula to obtain the overlap value MC between the calibration segment and the comparison segment; the specific calculation formula is:
[0077]
[0078] Among them, β1, β2, and β3 are respectively set weight constants, and β1>β2>β3>1; It can be seen from the formula that when the number of intersection keywords is greater and the corresponding importance value is greater, it means that the degree of overlap between the comparison fragment and the calibration fragment is greater, and the overlap value is greater; It should be noted that the data fragments with greater overlap are classified into the same batch. When processing this batch of data, it can reduce the frequent context switching and irrelevant data processing time during the data processing process, reduce the conflict during data integration, and improve the processing efficiency; Thus, the overlap value between each calibration fragment and each comparison fragment can be obtained;
[0079] Step 2: Calculate the time difference between the timestamp corresponding to each comparison segment and the current system time to obtain the interval length, and record it as T; it should be noted that the data segment with a larger interval length should be processed first; normalize the interval length T and the coincidence value MC and take their values, and analyze the values to obtain the batch value TM of the comparison segment. The specific calculation formula is:
[0080]
[0081] Among them, α1 and α2 are the set weight constants. It can be seen from the formula that when the interval length is longer and the overlap value is larger, the batch value is larger;
[0082] Step 3: Starting with the calibration fragment, sort the comparison fragments in descending order according to their corresponding batch values; starting with the calibration fragment, intercept the data fragments of a set data volume (the specific data volume here is set by the person skilled in the art according to actual needs) as the same batch of data;
[0083] Step 4: Repeat the above steps 1 to 3 until all the data segments in the category tag are processed in batches; thus, the data segments in each category tag can be processed in batches;
[0084] By comprehensively analyzing and sorting the timestamp differences and overlapping values of the compared fragments, and assigning reasonable processing priorities to different data fragments, frequent context switching can be avoided, thereby reducing the delay caused by irrelevant data processing and improving the overall data processing efficiency; at the same time, it avoids the situation of processing data fragments too early or too late, thereby improving the balance and efficiency of the overall processing.
[0085] Step 3: Based on the batch value, dynamically optimize and adjust each batch of data to maximize data processing efficiency; specifically:
[0086] Step 1: Treat each batch of data as an independent task, thereby obtaining several independent tasks and the batch values corresponding to each comparison fragment in the independent task, and summing them up to obtain the batch value of the independent task; thus, the batch value of each independent task can be obtained, and the independent task with the largest batch value is selected as the calibration task;
[0087] Step 2: Set each device to correspond to a line energy value, recorded as B; obtain the number of independent tasks to be processed for each device in real time, and record it as Y; retrieve the busy value FR of each device closest to the current time of the system in real time, and normalize it with the number of independent tasks to be processed Y and the line energy value B, and take their values. Analyze the values to obtain the thread value YR. The specific calculation formula is:
[0088]
[0089] Among them, λ1, λ2, and λ3 are respectively the set weight constants. It can be seen from the formula that when the busy value is smaller, the number of independent tasks to be processed is smaller, and the thread value is larger; when the line energy value of the device is smaller, the thread value is larger; thus, the thread values of several devices can be obtained, and the device with the largest thread value is selected as the calibration device, and the calibration task is assigned to the calibration device, thereby increasing the number of independent tasks to be processed of the calibration device by one;
[0090] Step 3: Whenever the device completes an independent task, the number of independent tasks to be processed corresponding to it is reduced by one, and a processing record is added, wherein the processing record includes the processing start time and the processing end time; the cumulative processing record of the device is obtained, and the processing start time and the processing end time in each processing record are calculated by time difference to obtain a single processing time, and the single processing time corresponding to each processing record is calculated by average to obtain the line energy value, and it is updated to step 2;
[0091] Step 4: Repeat steps 1 to 3 until all independent tasks are processed;
[0092] By continuously monitoring and adjusting equipment load and performance, and dynamically adjusting task allocation accordingly, we can maximize resource utilization, avoid processing bottlenecks and task backlogs, adaptively allocate tasks, optimize equipment selection, and ultimately improve the data processing capabilities and stability of the entire system, realizing a dynamic adjustment and optimization mechanism for financial data processing.
[0093] Step 4: After each batch of data is processed, the processing results of each device are merged according to their corresponding category tags to ensure the consistency of the original data and avoid data loss or duplicate processing; the specific merging refers to: merging the processing results belonging to the same category tag.
[0094] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.
Claims
1. A data processing method for financial accounting office equipment, characterized in that: The following steps are involved: Step 1: Analyze the current business status in real time to determine whether optimization and adjustment are needed. If necessary, execute Step 2. Step 2: The financial data is fragmented and batched to obtain several batches of data, where each batch of data includes several data fragments arranged in sequence, except for the first-ranked data fragment, and each data fragment corresponds to a batch value; Step 3: Dynamically optimize and adjust each batch of data based on the batch value. The specific adjustment steps are: S101: Treat each batch of data as an independent task, thereby obtaining a number of independent tasks and batch values corresponding to each data segment in the independent task, and summing them up to obtain the batch value of the independent task; thereby obtaining the batch value of each independent task, and selecting the independent task with the largest batch value as the calibration task; S102: Set each device to correspond to a line energy value, recorded as B; obtain the number of independent tasks to be processed of each device in real time, recorded as Y; retrieve the busy value FR of each device closest to the current time of the system in real time, and normalize it with the number of independent tasks to be processed Y and the line energy value B, and take their values, analyze the values to obtain the thread value YR, and the specific calculation formula is: ; Where λ1, λ2, and λ3 are respectively set weight constants; thus, the thread values of several devices can be obtained, and the device with the largest thread value is selected as the calibration device, and the calibration task is assigned to the calibration device, thereby increasing the number of independent tasks to be processed of the calibration device by one; S103: Whenever the device completes an independent task, the number of independent tasks to be processed corresponding to it is reduced by one, and a processing record is added, wherein the processing record includes the processing start time and the processing end time; the cumulative processing record of the device is obtained, and the processing start time and the processing end time in each processing record are calculated by time difference to obtain a single processing time, and the single processing time corresponding to each processing record is calculated by average to obtain a line energy value, and the line energy value is updated to S102; S104: Repeat S101 to S103 until all independent tasks are processed; Step 4: After each batch of data is processed, the processing results of each device are merged according to their corresponding category labels.
2. A data processing method for financial accounting office equipment according to claim 1, characterized in that: The specific steps to determine whether optimization adjustment is needed are as follows: S201: acquiring the amount of data to be processed, the data processing speed, and the bandwidth occupancy rate of each device in real time, normalizing them and obtaining their values, and performing formula calculation and analysis on the values to obtain a load value; S202: constructing a two-dimensional rectangular coordinate system with time as the horizontal coordinate and load value as the vertical coordinate, inputting the load value into the coordinate axis according to its corresponding collection time, and recording the position of the load value in the coordinate axis as the load point, and connecting each load point in sequence with a smooth curve to obtain a curve graph of load value changing with time; S203: Perform fine-grained analysis on the load state of the device according to the load value versus time curve to obtain a busy value; thereby obtaining the busy value of each device; S204: Compare and analyze the busy value of each device with the set busy interval. When the busy value is greater than the set busy interval, the device corresponding to the busy value is recorded as a highly busy device; when the busy value is within the set busy interval, the device corresponding to the busy value is recorded as a moderately busy device; when the busy value is less than the set busy interval, the device corresponding to the busy value is recorded as a lowly busy device; The number of highly busy devices, medium busy devices and low busy devices are counted respectively and recorded as A1, A2 and A3 respectively; if A3 ≥ A1 + A2, no optimization adjustment is required; Otherwise, execute Step 2.
3. A data processing method for financial accounting office equipment according to claim 2, characterized in that: The specific steps for fine-grained analysis of the load status of the device based on the load value over time curve are as follows: Draw a tangent line at each load point, take the derivative of the tangent line to obtain the load derivative at each load point, sum the load derivatives greater than zero to obtain the load increase degree, and sum the load derivatives less than zero and take the absolute value to obtain the load reduction degree; The load value, load increase degree, and load reduction degree of the device at each collection time are normalized and taken as the numerical value, and the busy value is obtained by numerical analysis; thus, the busy value of each device can be obtained.
4. A data processing method for financial accounting office equipment according to claim 1, characterized in that: The specific steps of sharding are as follows: The financial data is segmented according to a specific byte size to obtain a number of data segments, keywords of each data segment are extracted, and the importance score of each keyword in the data segment is identified using TF-IDF; thereby, the keywords in each data segment and the importance scores corresponding to the keywords can be obtained; Suppose there are several categories, each category corresponds to a label, and each category label corresponds to several keywords; Extract keywords from the data segments and compare them with keywords corresponding to the category tags to obtain intersection keywords and union keywords, and count the number of intersection keywords and union keywords respectively; Retrieve the importance scores of each intersection keyword in the data segment, and sum them up to obtain the total importance score; compare and analyze the importance scores of each intersection keyword with the set score range to obtain the cumulative number of high intersections, medium intersections and low intersections; The number of intersection keywords, the number of union keywords, the cumulative number of high intersections, the cumulative number of medium intersections, the cumulative number of low intersections and the important total score are normalized and their values are taken, and the values are analyzed by formula calculation to obtain the score value; thereby, the score value between the data fragment and each category of target tags can be obtained, and the category tag with the largest score value is selected as the target tag of the data fragment, and the data fragment is classified into the target tag; thereby, all data fragments are classified into each category of target tags.
5. A data processing method for financial accounting office equipment according to claim 4, characterized in that: The specific steps of batch processing are as follows: S501: Select any category tag and the data segments it includes, and use the data segment whose timestamp is farthest from the current system as the calibration segment, and use the other data segments as the comparison segments; obtain the overlap value by analyzing the overlap degree between the calibration segment and the comparison segment, thereby obtaining the overlap value between each calibration segment and each comparison segment; S502: Calculate the time difference between the timestamp corresponding to each comparison segment and the current time of the system to obtain the interval duration, normalize the interval duration and the coincidence value and take their values, and analyze the values to obtain the batch values of the comparison segment; S503: starting with the calibration segment, sorting the comparison segments in descending order according to their corresponding batch values; starting with the calibration segment, intercepting a set amount of data segments as the same batch of data; S504: Repeat the above S501 to S503 until all the data segments in the category tag are processed in batches; thus, the data segments in each category tag can be processed in batches.
6. A data processing method for financial accounting office equipment according to claim 5, characterized in that: The specific steps of analyzing the overlap degree between the calibration fragment and the comparison fragment to obtain the overlap value are as follows: Retrieving keywords between the calibration segment and the comparison segment and the importance scores corresponding to the keywords, comparing the keywords of the two to obtain intersection keywords and union keywords, and counting the number of intersection keywords and union keywords; Then, the importance scores of the union keywords in the calibration segment and the comparison segment are averaged to obtain the importance value, thereby obtaining the importance value of each intersection keyword, and the sum of them is calculated to obtain the total importance value; Compare and analyze the importance value of the intersection keyword with the set importance interval to obtain the cumulative number of heavy correlation, medium correlation and light correlation; The intersection keywords, the number of union keywords, the cumulative number of heavily correlated, the cumulative number of moderately correlated, the cumulative number of lightly correlated and the total important value are normalized and their values are taken, and the values are calculated formulaically to obtain the overlap value between the calibration fragment and the comparison fragment.
Citation Information
Patent Citations
Multi-thread data fragment parallel method
CN117271137A
Data fragmentation and table division autonomous extension system and method
CN118606295A