Project audit management system and method based on big data
Through precise analysis and scientific allocation of different storage nodes and data characteristics, the problem of large deviation in data storage time difference in project audit data management is solved, and the balanced and efficient management of data storage is achieved.
Patent Information
- Application Number
- CN202510376741.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2025-04-25
- Estimated Expiration
- 2045-03-28
AI Technical Summary
The existing technology fails to effectively analyze the characteristics of different storage nodes and data in project audit data management, resulting in large deviations in time when data storage, resulting in data confusion and low management quality.
By accurately analyzing the storage characteristics of mechanical, solid-state and tape storage nodes, the data is scientifically and reasonably allocated to the most suitable storage nodes based on the type and capacity of the project data, and the storage process is monitored in real time, and the CPU proportion utilization rate is adjusted to ensure storage balance.
It realizes accurate allocation of different types of data, gives full play to the advantages of each storage node, reduces storage time difference, ensures the consistency of time characteristics of storage processes, and improves the efficiency and quality of data storage processing.
Smart Images

Figure CN119887126A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of project data management, and in particular to a project audit management system and method based on big data. Background Art
[0002] Project audit data management refers to the management of a series of activities such as collection, storage, processing, analysis and utilization of data involved in the project audit process, aiming to ensure the accuracy, integrity, security and availability of data, and provide strong support for project audit work. During a single project audit, a large amount of project audit data of different types will be generated.
[0003] The application with announcement number CN115759423B discloses a project audit management system and method based on big data, the system includes a change priority optimization module and an audit management module, the change priority optimization module analyzes the change in the construction period of the engineering change area corresponding to each element in the first array before and after the engineering change, optimizes the change priority of each element in the first array according to the analysis result, and obtains the optimized first array; the audit management module audits the project according to the optimization result of the change priority corresponding to each element in the first array in the change priority optimization module. The present invention manages the change priority of the engineering change area, and at the same time issues an early warning for the unreasonable part in the project change application materials, so as to realize the effective management of the project change application materials.
[0004] Project auditing is related to the data management process. Generally, based on specific storage nodes, a large amount of project data generated by a specified batch is optimized for storage, and the storage process is optimized based on the storage rate generated during the storage process. However, the original storage processing method did not analyze the characteristics of different storage nodes and different data. As a result, when the corresponding batches of data are stored, there will be a large deviation in the time difference for different storage nodes to complete the storage task. This will cause different storage nodes to process different data batches, resulting in confusion in the project audit data and low overall data management quality. Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention provides a project audit management system and method based on big data, which solves the problem that when project audit data is managed, the characteristics of different storage nodes and different data are not analyzed, which leads to confusion in project audit data.
[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A project audit management method based on big data, comprising the following steps:
[0007] Step 1: Perform initial storage process management on several groups of project data generated in a specified batch. According to the multiple groups of storage nodes preset in the audit system, perform initial determination on the storage characteristics of different storage nodes. Based on the determined storage characteristics, perform standard process allocation for different types of project data. According to the standard process, different types of project data are associated and allocated to different storage nodes. Different storage nodes perform storage processing on different types of project data.
[0008] The specific method for initially determining the storage characteristics of different storage nodes is as follows:
[0009] S11, confirming multiple groups of storage nodes preset in the audit system, and determining the classification of each group of storage nodes, wherein the classification includes mechanical storage nodes, solid-state storage nodes, and tape storage nodes, and after the corresponding storage nodes have completed the classification determination, classifying and marking the designated storage nodes, wherein the mechanical storage nodes correspond to classification mark A, the solid-state storage nodes correspond to classification mark B, and the tape storage nodes correspond to classification mark C;
[0010] S12. If the corresponding storage node belongs to a mechanical storage node with classification mark A: determine a set of tracing cycles, where the tracing cycle is a preset cycle, process several sets of storage rates generated within the tracing cycle, average several sets of storage rates generated in the single-path data storage process, confirm the single-path storage rate characteristics, and then average several sets of storage rates generated in the multi-path data storage process, confirm the multi-path storage rate characteristics, and use the confirmed single-path storage rate characteristics and multi-path storage rate characteristics as the storage characteristics of this storage node;
[0011] If the corresponding storage node belongs to a solid-state storage node with classification mark B: then determine a set of tracing cycles, where the tracing cycle is a preset cycle, average the storage rates of several groups generated by the storage node within the tracing cycle, confirm the rate average, and use the confirmed rate average as the storage feature of this storage node;
[0012] If the corresponding storage node belongs to a tape storage node with classification mark C: similarly determine a set of tracing cycles, where the tracing cycle is a preset cycle, confirm several groups of storage rates generated by the storage node in the storage process of single-path data within the tracing cycle, average the confirmed several groups of storage rates, confirm the rate average, and use the confirmed rate average as the storage feature of this storage node;
[0013] The specific method of associating different types of project data to different storage nodes is as follows:
[0014] S13, identifying the number G of data formats within a single group of different types of project data, if G=1, then marking this type of project data as single-path data, if G≥2, then marking this type of project data as multi-path data;
[0015] S14, based on the calibrated single-path data capacity, sorting the calibrated single-path data in ascending order according to the capacity value to confirm the single-path data sequence, synchronously confirming the capacity of the multi-path data, and synchronously sorting the multi-path data in ascending order according to the capacity value to confirm the multi-path data sequence;
[0016] S15, for the mechanical storage node with classification mark A, data is selected from the single-path data sequence from back to front first, and when the single-path data sequence is selected, data is selected from the multi-path data sequence from front to back;
[0017] For the solid-state storage node with classification mark B, data is selected from the multi-path data sequence from the back to the front. When the multi-path data sequence is selected, the remaining data is randomly selected from the single-path data sequence.
[0018] For the tape storage node with classification mark C, data is selected from the single path data sequence from the front to the back;
[0019] S16, randomly executing several selection processes, each of which distributes all data in the single-path data sequence and the multi-path data sequence to different storage nodes, and confirming the time characteristic value associated with each group of selection processes, specifically in the following manner:
[0020] Based on the storage characteristics associated with the corresponding storage node, the time characteristic value associated with the corresponding node is confirmed to identify whether the corresponding storage node is a mechanical storage node with classification mark A:
[0021] If yes, record the total single-path data capacity and multi-path data capacity allocated to this storage node, and mark the recorded total single-path data capacity as R 单 , the total capacity of the recorded multipath data is calibrated as R 多 , the single-path storage rate characteristic confirmed by this storage node is calibrated as V 单 , the confirmed multi-path storage rate characteristic is calibrated as V 多 , using: R 单 ÷V 单 =T 单 and R 多 ÷V 多 =T 多 Determine the time characteristics associated with different types of data, and then 单 and T多 Perform summation to determine the time characteristic value ST of this storage node;
[0022] If not, then record the total capacity Rz of all data allocated to this storage node, then confirm the storage feature Tz associated with this storage node, and use Rz÷Tz=JT to confirm the time feature value JT belonging to this storage node;
[0023] S17, processing different time feature values associated with different storage nodes in different selection processes: performing variance processing on multiple groups of time feature values associated with different storage nodes of a single group of selection processes, confirming the feature variance belonging to this selection process, and then selecting the same processing method for other selection processes to confirm the feature variance of this selection process;
[0024] A minimum value is selected from different feature variances associated with multiple groups of different selection processes, and the selection process corresponding to the minimum value is recorded as a standard process. According to the standard process, different types of project data are associated and allocated to different storage nodes;
[0025] Step 2: After the specific allocation of different types of project data is completed, the storage process associated with each storage node is monitored in real time, and based on the real-time monitored storage rate, the CPU utilization rate associated with different storage nodes is readjusted. The specific sub-steps are:
[0026] S21. Based on the storage characteristics confirmed by the corresponding storage node, lock the monitoring interval of this storage node and propose the storage characteristics as V i , where i represents different storage nodes, and the locked monitoring interval is [V i -Y1, V i +Y1], where Y1 is a preset value. If the corresponding storage node is a mechanical storage node with classification mark A, there are two sets of monitoring intervals for such storage nodes;
[0027] S22. Based on the monitoring interval confirmed by the corresponding storage node, the storage rate associated with different storage nodes in the storage process is monitored in real time, and based on the specific results of the real-time monitoring, the CPU utilization ratio of the storage node is readjusted:
[0028] If the monitored real-time storage rate Vs∉[V i -Y1, V i +Y1], high and low adjustment is performed. When Vs<V i -Y1, the CPU utilization of the corresponding storage node is increased to the real-time storage rate Vs∈[V i -Y1, V i +Y1], if Vs>V i+Y1, the CPU utilization of the corresponding storage node is reduced to the real-time storage rate Vs∈[V i -Y1, V i +Y1];
[0029] Step 3: After all the groups of project data associated in this batch are stored, an index sequence table of this batch is established based on the storage duration associated with the corresponding project data. The specific sub-steps are:
[0030] S31. Confirm the storage duration of several groups of project data in this batch, and mark the storage duration associated with different project data as SJ. k , where k represents different project data;
[0031] S32, based on different storage durations SJ k Different specific values of will be sorted from small to large according to the corresponding storage duration SJ k Sort the associated different project data and establish the index sequence table of this batch;
[0032] S33. When indexing the project data of this batch, indexing is performed step by step from front to back according to this index sequence table.
[0033] Preferably, a project audit management system based on big data includes:
[0034] The node feature determination end, based on the multiple groups of storage nodes preset in this audit system, prioritizes the classification of the corresponding storage nodes, and then processes the historical completion data of storage nodes of different classifications in different ways to preliminarily determine the storage features of different storage nodes;
[0035] The project data distribution end determines the standard process for distributing different types of project data based on the different storage characteristics determined by different storage nodes, associates different types of project data with different storage nodes according to the standard process, and stores and processes different types of project data by different storage nodes;
[0036] The CPU utilization adjustment end monitors the storage process associated with each storage node in real time after the specific allocation of different types of project data is completed, and readjusts the CPU utilization associated with different storage nodes based on the real-time monitored storage rate;
[0037] At the index sequence table construction end, after the storage of several groups of project data associated in this batch is completed, the index sequence table of this batch is established based on the storage time associated with the corresponding project data and in ascending order of storage time.
[0038] The present invention provides a project audit management system and method based on big data. Compared with the prior art, it has the following beneficial effects:
[0039] The present invention accurately analyzes and determines the storage characteristics of three types of storage nodes: mechanical, solid-state and magnetic tape. It can accurately allocate different types of data to the most suitable storage node according to the type of project data (single-path or multi-path data) and its capacity with scientific and reasonable allocation logic. This allocation method not only gives full play to the advantages of each storage node, but also makes the storage time difference of each storage node when storing different types of data smaller, ensures the relative consistency of the storage process in time characteristics, effectively avoids the conflict of two batches of project data stored in the same time period, and greatly improves the overall effect and efficiency of data storage and processing.
[0040] During the data storage process, the storage process of each storage node is monitored in real time, and the CPU utilization of the storage node is flexibly adjusted according to the preset monitoring interval and storage characteristics. When the real-time storage rate monitored deviates from the normal range, it can be adjusted up or down in time to ensure that the storage rate is always within a reasonable range, thereby achieving effective control of storage balance, greatly reducing data storage time, and further improving the quality and efficiency of storage processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a schematic diagram of the process of the present invention;
[0042] Figure 2 It is a schematic diagram of the principle framework of the present invention. DETAILED DESCRIPTION
[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] See also Figure 1 , this application provides a project audit management method based on big data, including the following steps:
[0045] Step 1: Perform initial storage process management on several groups of project data generated in the specified batch. According to the multiple groups of storage nodes preset in this audit system, perform initial determination on the storage characteristics of different storage nodes. Based on the determined storage characteristics, allocate and determine the standard process for different types of project data. According to the standard process, different types of project data are associated and allocated to different storage nodes. Different storage nodes store and process different types of project data. Specifically, in this audit system, there are three different types of corresponding storage nodes, namely mechanical storage nodes, solid-state storage nodes and tape storage nodes. Each different storage node has different storage characteristics:
[0046] Its mechanical storage nodes have strong single-path data read and write performance and fast read and write rates, but for multi-path data, its read and write rates are slow;
[0047] Solid-state storage nodes have fast read and write speeds for both single-path and multi-path data, and their overall performance is strong.
[0048] Tape storage nodes can only process the storage process of single-path data. They cannot execute the associated storage process for multi-path data, and the read and write rates for single-path data are low.
[0049] Single-path data: This type of data is simply a single-set format data, and only a single set of paths is needed to realize the reading and writing process of this type of data, that is: pure text data, pure code data or pure video data, pure audio data, etc.
[0050] Multi-path data, which is a combination of multiple sets of data in different formats. For the read and write storage of such multi-path data, the corresponding storage nodes need to execute multiple sets of different storage paths to complete the complete storage process of such multi-path data.
[0051] The specific method for initially determining the storage characteristics of different storage nodes is as follows:
[0052] S11, confirming multiple groups of storage nodes preset in the audit system, and determining the classification of each group of storage nodes, wherein the classification includes mechanical storage nodes, solid-state storage nodes, and tape storage nodes, and after the corresponding storage nodes have completed the classification determination, classifying and marking the designated storage nodes, wherein the mechanical storage nodes correspond to classification mark A, the solid-state storage nodes correspond to classification mark B, and the tape storage nodes correspond to classification mark C;
[0053] S12. If the corresponding storage node belongs to a mechanical storage node with classification mark A (that is, this storage node belongs to a mechanical storage node): then determine a set of traceability cycles, the traceability cycle is a preset cycle, which is prepared in advance by relevant operators based on experience, process several groups of storage rates generated within the traceability cycle, average several groups of storage rates generated in the single-path data storage process, confirm the single-path storage rate characteristics, and then average several groups of storage rates generated in the storage process of multi-path data to confirm the multi-path storage rate characteristics, and use the confirmed single-path storage rate characteristics and multi-path storage rate characteristics as the storage characteristics of this storage node, the traceability cycle is a historical cycle, and the mechanical storage nodes, for single-path data or multi-path data, have different storage rates associated, so for such storage nodes, it is necessary to determine the storage characteristics related to the corresponding storage nodes based on the storage conditions of different data;
[0054] If the corresponding storage node belongs to a solid-state storage node with classification mark B (that is, this storage node belongs to a solid-state storage node): then determine a set of tracing cycles (this tracing cycle is the same as the tracing cycle mentioned in the previous paragraph), and its tracing cycle is a preset cycle. Perform average processing on several groups of storage rates generated by this storage node within the tracing cycle, confirm the rate average, and use the confirmed rate average as the storage feature of this storage node (for solid-state storage nodes, the average speed generated for single-path data or multi-path data is relatively consistent, so a unified confirmation method can be used to determine the storage rate associated with the corresponding storage node, thereby determining the storage feature associated with the corresponding node);
[0055] If the corresponding storage node belongs to a tape storage node with classification mark C: similarly determine a set of tracing cycles, where the tracing cycle is a preset cycle, confirm several groups of storage rates generated by the storage node in the storage process of single-path data within the tracing cycle, average the confirmed several groups of storage rates, confirm the rate average, and use the confirmed rate average as the storage feature of this storage node;
[0056] The specific method of associating different types of project data to different storage nodes is as follows:
[0057] S13, identifying the number G of data formats within a single group of different types of project data, if G=1, then marking this type of project data as single-path data, if G≥2, then marking this type of project data as multi-path data;
[0058] S14, based on the calibrated single-path data capacity, sorting the calibrated single-path data in ascending order according to the capacity value to confirm the single-path data sequence, synchronously confirming the capacity of the multi-path data, and synchronously sorting the multi-path data in ascending order according to the capacity value to confirm the multi-path data sequence;
[0059] S15, for the mechanical storage node with classification mark A, data is selected from the single-path data sequence from back to front first, and when the single-path data sequence is selected, data is selected from the multi-path data sequence from front to back;
[0060] For the solid-state storage node with classification mark B, data is selected from the multi-path data sequence from the back to the front. When the multi-path data sequence is selected, the remaining data is randomly selected from the single-path data sequence.
[0061] For the tape storage node with classification mark C, data is selected from the single path data sequence from the front to the back;
[0062] S16. Randomly execute several selection processes, each of which distributes all data in the single-path data sequence and the multi-path data sequence to different storage nodes, and confirms the time characteristics associated with each group of selection processes:
[0063] S161: Based on the storage feature associated with the corresponding storage node, confirm the time feature value associated with the corresponding node, and identify whether the corresponding storage node is a mechanical storage node with a classification mark A:
[0064] If yes, record the total single-path data capacity and multi-path data capacity allocated to this storage node, and mark the recorded total single-path data capacity as R 单 , the total capacity of the recorded multipath data is calibrated as R 多 , the single-path storage rate characteristic confirmed by this storage node is calibrated as V 单 , the confirmed multi-path storage rate characteristic is calibrated as V 多 , using: R 单 ÷V 单 =T 单 and R 多 ÷V 多 =T 多 Determine the time characteristics associated with different types of data, and then 单 and T 多 Perform summation to determine the time characteristic value ST of this storage node;
[0065] If not, then record the total capacity Rz of all data allocated to this storage node, then confirm the storage feature Tz associated with this storage node, and use Rz÷Tz=JT to confirm the time feature value JT belonging to this storage node;
[0066] S17, processing different time feature values associated with different storage nodes in different selection processes: performing variance processing on multiple groups of time feature values associated with different storage nodes of a single group of selection processes, confirming the feature variance belonging to this selection process, and then selecting the same processing method for other selection processes to confirm the feature variance of this selection process;
[0067] A minimum value is selected from different feature variances associated with multiple groups of different selection processes, and the selection process corresponding to the minimum value is recorded as a standard process. According to the standard process, different types of project data are associated and allocated to different storage nodes;
[0068] Specifically, this processing method allocates different types of data to storage nodes with different storage characteristics based on the storage characteristics of different storage nodes and the specific types of different data. Based on the determined specific data characteristics and specific allocation logic, the corresponding storage nodes can effectively store different types of data. When each different storage node stores different types of data, the storage time associated with it is relatively consistent in difference, which can achieve better data allocation effect. Before the next batch of data arrives, there will be no situation where the project data of the previous and next batches are stored in the same time period, which can effectively improve the storage processing effect of its data and ensure that different storage processes are relatively consistent in time characteristics.
[0069] Step 2: After the specific allocation of different types of project data is completed, the storage process associated with each storage node is monitored in real time, and based on the real-time monitored storage rate, the CPU utilization rate associated with different storage nodes is readjusted, wherein the specific sub-steps of the readjustment are:
[0070] S21. Based on the storage characteristics confirmed by the corresponding storage node, lock the monitoring interval of this storage node and propose the storage characteristics as V i , where i represents different storage nodes, and the locked monitoring interval is [V i -Y1, V i +Y1], where Y1 is a preset value, and its specific value is determined by the operator based on experience. If the corresponding storage node is a mechanical storage node with classification mark A, there are two sets of monitoring intervals for such storage nodes;
[0071] S22. Based on the monitoring interval confirmed by the corresponding storage node, the storage rate associated with different storage nodes in the storage process is monitored in real time, and based on the specific results of the real-time monitoring, the CPU utilization ratio of the storage node is readjusted:
[0072] If the monitored real-time storage rate Vs∉[V i -Y1, V i +Y1], high and low adjustment is performed. When Vs<V i -Y1, the CPU utilization of the corresponding storage node is increased to the real-time storage rate Vs∈[V i -Y1, V i +Y1], if Vs>V i +Y1, the CPU utilization of the corresponding storage node is reduced to the real-time storage rate Vs∈[V i -Y1, V i +Y1];
[0073] Specifically, by adopting this processing method, the storage process of each storage node can be fully and effectively monitored in real time. Based on the specific results of real-time monitoring, the associated storage rate can be controlled to achieve effective control of storage balance, to fully reduce the corresponding storage time, and to achieve better storage processing effects.
[0074] Step 3: After all the project data associated with the batch are stored, an index sequence table of the batch is established based on the storage duration associated with the corresponding project data. When the data of the batch is indexed later, the indexing is performed step by step according to the sequence table, so as to achieve a better indexing effect.
[0075] The specific sub-steps of establishing the index sequence table are:
[0076] S31. Confirm the storage duration of several groups of project data in this batch, and mark the storage duration associated with different project data as SJ. k , where k represents different project data;
[0077] S32, based on different storage durations SJ k Different specific values of will be sorted from small to large according to the corresponding storage duration SJ k Sort the associated different project data and establish the index sequence table of this batch;
[0078] S33. When indexing the project data of this batch, indexing is performed step by step from front to back according to this index sequence table. This indexing method can ensure that the corresponding project data is output step by step, and will not cause the corresponding personnel to wait for a long time, so that the corresponding batch of data can be output step by step. This indexing effect is better and can fully reduce the waiting time for indexing.
[0079] Combination Figure 2 , a project audit management system based on big data, including:
[0080] The node feature determination end, based on the multiple groups of storage nodes preset in this audit system, prioritizes the classification of the corresponding storage nodes, and then processes the historical completion data of storage nodes of different classifications in different ways to preliminarily determine the storage features of different storage nodes;
[0081] The project data distribution end determines the standard process for distributing different types of project data based on the different storage characteristics determined by different storage nodes, associates different types of project data with different storage nodes according to the standard process, and stores and processes different types of project data by different storage nodes;
[0082] The CPU utilization adjustment end monitors the storage process associated with each storage node in real time after the specific allocation of different types of project data is completed, and readjusts the CPU utilization associated with different storage nodes based on the real-time monitored storage rate;
[0083] At the index sequence table construction end, after the storage of several groups of project data associated in this batch is completed, the index sequence table of this batch is established based on the storage time associated with the corresponding project data and in ascending order of storage time.
[0084] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.
[0085] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.
Claims
1. A project audit management method based on big data, characterized in that: The following steps are involved: Step 1: Perform initial storage process management on several groups of project data generated in a specified batch. According to the multiple groups of storage nodes preset in the audit system, perform initial determination on the storage characteristics of different storage nodes. Based on the determined storage characteristics, perform standard process allocation for different types of project data. According to the standard process, different types of project data are associated and allocated to different storage nodes. Different storage nodes perform storage processing on different types of project data. Step 2: After the specific allocation of different types of project data is completed, the storage process associated with each storage node is monitored in real time, and based on the real-time monitored storage rate, the CPU utilization rate associated with different storage nodes is readjusted; Step 3: After all the groups of project data associated in this batch are stored, an index sequence table of this batch is established based on the storage duration associated with the corresponding project data.
2. According to the project audit management method based on big data in claim 1, it is characterized in that: In step 1, the specific method of initially determining the storage characteristics of different storage nodes is: S11, confirming multiple groups of storage nodes preset in the audit system, and determining the classification of each group of storage nodes, wherein the classification includes mechanical storage nodes, solid-state storage nodes, and tape storage nodes, and after the corresponding storage nodes have completed the classification determination, classifying and marking the designated storage nodes, wherein the mechanical storage nodes correspond to classification mark A, the solid-state storage nodes correspond to classification mark B, and the tape storage nodes correspond to classification mark C; S12. If the corresponding storage node belongs to a mechanical storage node with classification mark A: determine a set of tracing cycles, where the tracing cycle is a preset cycle, process several sets of storage rates generated within the tracing cycle, average several sets of storage rates generated in the single-path data storage process, confirm the single-path storage rate characteristics, and then average several sets of storage rates generated in the multi-path data storage process, confirm the multi-path storage rate characteristics, and use the confirmed single-path storage rate characteristics and multi-path storage rate characteristics as the storage characteristics of this storage node; If the corresponding storage node belongs to a solid-state storage node with classification mark B: then determine a set of tracing cycles, where the tracing cycle is a preset cycle, average the storage rates of several groups generated by the storage node within the tracing cycle, confirm the rate average, and use the confirmed rate average as the storage feature of this storage node; If the corresponding storage node belongs to a tape storage node with classification mark C: also determine a set of tracing cycles, where the tracing cycle is a preset cycle, confirm several groups of storage rates generated by this storage node in the storage process of single-path data within the tracing cycle, and average the confirmed groups of storage rates to confirm the rate average, and use the confirmed rate average as the storage feature of this storage node.
3. According to the project audit management method based on big data in claim 1, it is characterized in that: In step 1, the specific method of associating and allocating different types of project data to different storage nodes is: S13, identifying the number G of data formats within a single group of different types of project data, if G=1, then marking this type of project data as single-path data, if G≥2, then marking this type of project data as multi-path data; S14, based on the calibrated single-path data capacity, sorting the calibrated single-path data in ascending order according to the capacity value to confirm the single-path data sequence, synchronously confirming the capacity of the multi-path data, and synchronously sorting the multi-path data in ascending order according to the capacity value to confirm the multi-path data sequence; S15, for the mechanical storage node with classification mark A, data is selected from the single-path data sequence from back to front first, and when the single-path data sequence is selected, data is selected from the multi-path data sequence from front to back; For the solid-state storage node with classification mark B, data is selected from the multi-path data sequence from the back to the front. When the multi-path data sequence is selected, the remaining data is randomly selected from the single-path data sequence. For the tape storage node with classification mark C, data is selected from the single path data sequence from the front to the back; S16, randomly executing a number of selection processes, each of which distributes all data in the single-path data sequence and the multi-path data sequence to different storage nodes, and confirming the time characteristic value associated with each group of selection processes; S17, processing different time feature values associated with different storage nodes in different selection processes: performing variance processing on multiple groups of time feature values associated with different storage nodes of a single group of selection processes, confirming the feature variance belonging to this selection process, and then selecting the same processing method for other selection processes to confirm the feature variance of this selection process; From different feature variances associated with multiple groups of different selection processes, a minimum value is selected, and the selection process corresponding to the minimum value is recorded as a standard process. Based on this standard process, different types of project data are associated and allocated to different storage nodes.
4. A project audit management method based on big data according to claim 3, characterized in that: In step S16, the specific method of confirming the time characteristic value associated with each group of selected processes is: Based on the storage characteristics associated with the corresponding storage node, the time characteristic value associated with the corresponding node is confirmed to identify whether the corresponding storage node is a mechanical storage node with classification mark A: If yes, record the total single-path data capacity and multi-path data capacity allocated to this storage node, and mark the recorded total single-path data capacity as R 单 , the total capacity of the recorded multipath data is calibrated as R 多 , the single-path storage rate characteristic confirmed by this storage node is calibrated as V 单 , the confirmed multi-path storage rate characteristic is calibrated as V 多 , using: R 单 ÷V 单 =T 单 and R 多 ÷V 多 =T 多 Determine the time characteristics associated with different types of data, and then 单 and T 多 Perform summation to determine the time characteristic value ST of this storage node; If not, then record the total capacity Rz of all data allocated to this storage node, and then confirm the storage characteristic Tz associated with this storage node, and use Rz÷Tz=JT to confirm the time characteristic value JT belonging to this storage node.
5. According to the project audit management method based on big data in claim 1, it is characterized in that: In step 2, the specific sub-steps for readjusting the CPU utilization ratios associated with different storage nodes are: S21. Based on the storage characteristics confirmed by the corresponding storage node, lock the monitoring interval of this storage node and propose the storage characteristics as V i , where i represents different storage nodes, and the locked monitoring interval is [V i -Y1, V i +Y1], where Y1 is a preset value. If the corresponding storage node is a mechanical storage node with classification mark A, there are two sets of monitoring intervals for such storage nodes; S22. Based on the monitoring interval confirmed by the corresponding storage node, the storage rate associated with different storage nodes in the storage process is monitored in real time, and based on the specific results of the real-time monitoring, the CPU utilization ratio of the storage node is readjusted: If the monitored real-time storage rate Vs∉[V i -Y1, V i +Y1], high and low adjustment is performed. When Vs<V i -Y1, the CPU utilization of the corresponding storage node is increased to the real-time storage rate Vs∈[V i -Y1, V i +Y1] stops.
6. A project audit management method based on big data according to claim 5, characterized in that: In step S22, if Vs>V i +Y1, the CPU utilization of the corresponding storage node is reduced to the real-time storage rate Vs∈[V i -Y1, V i +Y1] stops.
7. A project audit management method based on big data according to claim 1, characterized in that: In step 3, the specific sub-steps of establishing the index sequence table are: S31. Confirm the storage duration of several groups of project data in this batch, and mark the storage duration associated with different project data as SJ. k , where k represents different project data; S32, based on different storage durations SJ k Different specific values of will be sorted from small to large according to the corresponding storage duration SJ k Sort the associated different project data and establish the index sequence table of this batch; S33. When indexing the project data of this batch, indexing is performed step by step from front to back according to this index sequence table.
8. A project audit management system based on big data, which operates according to a project audit management method based on big data according to any one of claims 1 to 7, characterized in that: include: The node feature determination end, based on the multiple groups of storage nodes preset in this audit system, prioritizes the classification of the corresponding storage nodes, and then processes the historical completion data of storage nodes of different classifications in different ways to preliminarily determine the storage features of different storage nodes; The project data distribution end determines the standard process for distributing different types of project data based on the different storage characteristics determined by different storage nodes, associates different types of project data with different storage nodes according to the standard process, and stores and processes different types of project data by different storage nodes; The CPU utilization adjustment end monitors the storage process associated with each storage node in real time after the specific allocation of different types of project data is completed, and readjusts the CPU utilization associated with different storage nodes based on the real-time monitored storage rate; At the index sequence table construction end, after the storage of several groups of project data associated in this batch is completed, the index sequence table of this batch is established based on the storage time associated with the corresponding project data and in ascending order of storage time.
Citation Information
Patent Citations
A project audit management system and method based on big data
CN115759423B
Data management system and method based on hybrid storage, terminal and medium
CN109491618A
Data storage management method, system and device and computer readable storage medium
CN115543185A
Data storage method based on solid state disk and storage medium
CN119336277A
Digital country data storage system based on cloud computing
CN119576224A